mScriptBox Tutorial
Introduction to Tokens

Written by Pasmal
Published with permission

Table Of Contents

  1. Introduction
  2. What Are Tokens?
  3. Token Identifiers
    1. $gettok
    2. $deltok
    3. $istok
    4. $addtok
    5. $remtok
    6. $numtok
    7. $wildtok
    8. $reptok
    9. $puttok
    10. $matchtok
    11. $instok
    12. $findtok
  4. Tokenize
  5. Practical Example: Swear Kicker
  6. Practical Example: Userlist Viewer
  7. Practical Example: Dice Roller

1. Introduction  Back to Top

Tokens are a somewhat hard concept to grasp, but like most concepts once grasped it becomes very useful. You may want to try visualise what I am explaining, I found it very helpful when I first learnt tokens. Before you learn tokens you should have a general idea on what variables and parameters are.


2. What are Tokens?  Back to Top

In general terms, tokens are pieces of a whole. Tokens are separated by a common character (Abbreviated to C), which is sometimes called the separator. I can use several analogies to explain this concept, but I think only one is truly effective:
Bricks and Mortar make walls. The wall is the whole. The Bricks are the tokens. The Mortar is the Separator - it keeps each brick to itself. The table below tries to show this. The lines are the mortar, the separators. They keep the Bricks from touching each other, they keep the tokens from touching each other.

So each separate brick represents a single token.
Tokens don't have to be spearated by spaces.

Token Terms
Text
The whole. The complete piece of data that we wish to use tokens with.
Token
A piece. A part of the whole separated by a common character.
C
A common character (also known as the Separator). When we state the common character we use its ASCII value ($asc(C))
N
The N'th token (piece) of the text. N stands for Number.

First Example
We have some text (the whole):

A quick brown fox jumped over the lazy dog

Now for this example our common character that separates each token will be the space. The space characters ASCII value is 32.

What is the first token?

This is asking what is the first piece of text separated by a space. A quick look at the text should show that A is the first token.
What is the fourth token?
Once again a quick look at the text should show that fox is the fourth token.
How many tokens are in the text?
This is asking you how many times does the common character divide or 'chop up' the text. In our example we can see that the space (the C for our example) is used to seperate the text into 9 pieces.
Second Example
Once again we have some text (the whole):

Bob.Harry.Sally.Jerry.Jesse.Melanie.Alex.David

In this example the space is not our common character. Care to guess what is? Its the full stop. The ASCII value of a full stop is 46. To find this yourself type //echo -a $asc(.)

What is the first token?

This is asking what is the first piece of text seperated by a full stop. A quick look at the text should show that Bob is the first token. Note how I did not include the full stop in my answer.
What is the sixth token?
Once again a quick look at the text should show that Melanie is the sixth token, as it is the sixth piece of text seperated by a full stop.
How many tokens are in the text?
This is asking how many times the text is divided or 'chopped up' by the full stop. A quick count should show that there are 8 tokens.
Why did I do a second example?
Just to point out that the common character can be anything you want it to be. It does not have to be a space.
We use tokens because they allow us to easily view and enter information from large groups of information. You could add/remove a swear word to a swear word kicker. You could add/remove a channel from a swear word kicker. You could get the username from a nickname's IRC address. Another common use is in dice rolling scripts. Plus many many more things.


3. Token Identifier  Back to Top

mIRC comes with a large assortment of identifiers to help you with your tokening. The below is a non alphabetical list of them. We will start with the more commonly used identifiers and slowly work by way down to the least used (Or the least used in my opinion).
Before we start, lets set a few variables:

/set %names Bob.Harry.Sally.Jerry.Jesse.Melanie.Alex.David 
/set %sentance A quick brown fox jumped over the lazy dog 
NOTE: Don't include the colour when you type the set command! Its just to make things easier to read in this tutorial.


3a. $GetTok(Text,N,C)  Back to Top

Just before I explain this identifier, lets recap:
Token Terms
Text
The whole. The complete piece of data that we wish to use tokens with.
Token
A piece. A part of the whole separated by a common character.
C
A common character (also known as the Separator). When we state the common character we use its ASCII value ($asc(C))
N
The N'th token (piece) of the text. N stands for Number.
OK here goes. The $gettok identifier is used to return the value of a token in some text. I won't give you a practical application for it just yet, that will be done in a section purely for practical examples.

Example:

We want to know what the third token is in the %names variable. The common character for the %names variable is a full stop (ASCII value of 46). We would use $gettok(%names,3,46):
//echo -a $gettok(%names,3,46)
This should return Sally

If you specify 0 as your N, it will return the number of tokens in the text.


3b. $DelTok(Text,N,C)  Back to Top

This identifier is used to remove a token from some text. You must know the token's position in order to use this identifier - is it the first token?, the second token?, the eighteenth token? etc...

Example1:

We want to remove quick from the %sentance variable. It is the second token. We need to use $deltok(%sentance,2,32)
//echo -a $deltok(%sentance,2,32)
This should return a brown fox jumped over the lazy dog

Please be aware that this did not remove quick from the variable %sentance. If we wanted to do that (which we do not) we would type something like: //set %sentance $deltok(%sentance,2,32)

This identifier is used to remove a token from some text. This identifier differs from the $remtok identifier because you enter the token number (e.g. what place the token is in the text).

Example 2:

We want to delete the 4th token in %names (Jerry). The common character is a full stop, which has an ASCII value of 46.
//echo -a $deltok(%names,4,46) 
This should return Bob.Harry.Sally.Jesse.Melanie.Alex.David

We can also remove multiple tokens in one go by changing the N value. Lets say we wanted to remove tokens 2-5 (Harry.Sally.Jerry.Jesse).

//echo -a $deltok(%names,2-5,46) 


3c. $IsTok(Text,Token,C)  Back to Top

This identifier is used to tell if a token is in some text. If the entire token is in the text it will return $true, else it will return $false. Wildcards will not work in this.

Example:

We want to tell if Melanie is in %names. We need to use $istok(%names,Melanie,46) - remember that the common character for %names is a full stop, which has a ASCII value of 46.
//echo -a $istok(%names,Melanie,46)
This should return $true, as Melanie is in %names. Actually she needs to be more than just in %names, Melanie needs to be a token by herself. If %names was set to Bob Harry.Sally Jerry.Jesse Melanie.Alex David then things would be different (Note how I removed every second stop). If this was the case the above echo would return $false.


3d. $AddTok(Text,Token,C)  Back to Top

This identifier is used to add a token to some text. If the token you are trying to add already exists in the text, then another entry will NOT be created - which saves you placing error checking code in your script :)

Example:

We want to add Aaron to %names. We are using the full stop (ASCII value 46) as our common character.
//echo -a $addtok(%names,Aaron,46)
It should return something like Bob.Harry.Sally.Jerry.Jesse.Melanie.Alex.David.Aaron

Now try it again, this time lets try to add Jesse to the list (Note he is already in the list).

//echo -a $addtok(%names,Jesse,46)
It should return Bob.Harry.Sally.Jerry.Jesse.Melanie.Alex.David


3e. $RemTok(Text,Token,N,C)  Back to Top

This identifier is used to remove a specified token from some text. This identifier differs from the $deltok identifier because you enter the actual token you want to remove, not just the place of the token in the text. You can also choose which matching token to remove if there is more than one match in the list.

Example:

We want to remove quick from a quick brown fox jumped over the lazy dog.
//echo -a $remtok(%names,quick,1,32) 
This should return a brown fox jumped over the lazy dog

As you may have picked up now, $addtok and $remtok are really handy when used together. You can let the person using the script add/remove stuff at ease. By this I mean they could edit a swear word list, a channel list, a nickname list - whatever. Its all up to your imagination.


3f. $NumTok(Text,C)   Back to Top

This identifier is used to return the total number of tokens in the text. This performs the same function as $gettok(text,0,C).

Example:

We want to find out how many people (tokens) we have in %names.
//echo -a $numtok(%names,46) 
This should return 8


3g. $WildTok(Tokens,Wildstring,N,C)  Back to Top

This identifier is used to find and return a token based on a wildcard that you enter. You can choose the match number to return (N). Wildcards accepted include * and ?.

Example:

We know someone in %names name starts with M. Use $wildtok to try find that name.
//echo -a $wildtok(%names,M*,1,46) 
This should return Melanie.

Also, we can only remember the last part of someone's name, lly someone. We also know that the name is 5 letters long. Once again we can use $wildtok to find a match.

//echo -a $wildtok(%names,??lly,1,46)
This should return Sally.

But what if we want to know the total number of matches so we can loop through them all? Use 0 as your N. For an example lets say we want to know how many people in %names have a as the second letter in their name.

//echo -a $wildtok(%names,?a*,0,46)
This should return 3.


3h. $RepTok(Text,Token,New Token,N,C)  Back to Top

This identifier is similar to $replace, but does have a few major differences. It is used to replace a token in some text with a new token - which you specify. Wildcards are not accepted.

Example:

We want to replace the word quick with the word fast in %sentance. The common character for %sentance is a space (ASCII value is 32).
//echo -a $reptok(%sentance,quick,fast,1,32)
This should return a fast brown fox jumped over the lazy dog.


3i. $PutTok(Text,New Token,N,C)  Back to Top

This identifier is similar to $reptok in the same way that $remtok and $deltok are similar to each other. Instead of specifying the actual token that you wish to overwrite with some new text, you specify its location/position in the text.

Example:

We want to replace the 3rd token in %sentance with the new token "green".
//echo -a $puttok(%sentance,green,3,32)
This should return a quick green fox jumped over the lazy dog.


3j. $MatchTok(Text,String,N,C)   Back to Top

This identifier is similar to $wildtok except that wildcards are not accepted. It performs a similar function: To find and return a token based on a search string that you enter.

Example:

We want to return the first word with a x in it in %sentance.
//echo -a $matchtok(%sentance,x,1,32)
This should return fox.

We can also return the total number of matches. Lets say we want to know the number of tokens in %sentance that contain "um".

//echo -a $matchtok(%sentance,um,0,32)
This should return 1, as there is only one match (jumped).


3k. $InsTok(Text,Token,N,C)  Back to Top

This identifier is used to insert a token into a specified location in the text. It differs to $puttok and $reptok because it does not overwrite any existing tokens, it simply 'pushes' the old tokens aside to make room for the new token (so to speak).

Example:

We want to add the world very just before the word quick in %sentance. Quick is the second token. So if we add very in as the second token, it should push quick over as the third token.
//echo -a $instok(%sentance,very,2,32)
This should return a very quick brown fox jumped over the lazy dog.


3l. $FindTok(Text,Token,N,C)   Back to Top

This identifier is used to find the location of a specified token. It differs to $wildtok because wildcards are not accepted, and it dosen't return the value of the token, it returns the location of the token in the text (e.g. what token number it is).

Example:

We want to find the location of the word (token) fox in %sentance.
//echo -a $findtok(%sentance,fox,1,32)
This should return 4.

Now why would we want to do this?
Well one possible reason could be that you want to use it in conjunction with $puttok.
If we wanted to change fox to donkey, we might use $puttok(%sentance,donkey,$findtok(%sentance,fox,1,32),32).
Of course you could just use $reptok: $reptok(%sentance,fox,donkey,1,32).


4. Tokenize  Back to Top

The /tokenize command is really neat :) It lets you turn a variable or whatever into $1, $2, etc.. ($N-M). You can choose the common character - the separator - so it dosen't always have to be a space. Its format is:

/tokenize C text
I found it particularly useful when dealing with data read from sockets (as my example shows). This example connects to www.7am.com and requests a page, http://www.7am.com/nzwires/. This page contains the news headlines for New Zealand. The tokenize command is used to make it easier to handle the information. This example is based off of my 7am News Ticker. Here is an example of how it works:

;This alias just opens a socket to www.7am.com - a news orientated web site.
alias 7am {
  IF ($sock(7am)) {
    RETURN
  }
  ELSE { sockopen 7am www.7am.com 80 }
} 
on *:SOCKOPEN:7am: { 
  IF ($sockerr) { echo 3 -a *** An error occured, script halted | RETURN } 
  ;If an error occured, we need to stop the script from going any further.
  ; If $sockerr is > 0, then an error has occured. 
  ;Once the socket is opened, we need to request the file that contains the news headers. 
  ;Unlucky for us, its in a HTML file so once we start receiving it
  ;we need to do a fair bit of parsing. 
  sockwrite -n 7am GET /nzwires/ HTTP/1.1 
  sockwrite -n 7am Host: www.7am.com 
  sockwrite -n 7am Connect: Keep-alive 
  sockwrite 7am $lf 
  window @7am 
  ;Creates a @window to display the headlines 
  aline @7am Requesting NZ News Headlines... 
} 
on *:SOCKREAD:7am: { 
  IF ($sockerr) { echo 3 -a *** An error occured, script halted | RETURN } 
  ;If an error occured, we need to stop the script from going any further.
    ;If $sockerr is > 0, then an error has occured. 
  sockread %7am 
  ;Sockread reads the data being sent to us from www.7am.com. 
  ;In our case its the HTML file that 
  ;contains the headlines of today's news in New Zealand. 
  tokenize 32 %7am 
  ;The tokenize command 'breaks up' this variable into $1 $2 etc...,
  ;using a space as the seperator. 
  ;This makes dealing with the data a lot easier,
  ;it removes the need for some $gettok, $deltok stuff. 
  ;However this dosen't mean that we never have to use them again. 
  IF ($1) { 
    ;The above if statement checks to see if $1 actually contains some information 
    IF ($wildtok($1-,a name="*">,1,60)) { 
      ;This $wildtok checks to see if there is a a name="*"> in the text. C is 60, or a <. 
      ;I choose to use < because we are dealing with an HTML page, and we want to find a specific 
      ;HTML tag, and HTML tags always start with a < - at least thats how I understand it. 
      SET %7am.temp.source $wildtok($1-,a name="*">,1,60) 
      ;Sets a variable to the HTML tag that matches our string of a name="*"> 
      SET %7am.temp.source $gettok(%7am.temp.source,2,34) 
      ;Sets the same variable to just the text inside the two speech marks. 
      ;We can use tokens to get this text, because we know that the text 
      ;inside the two speech marks ("*") can be the second token of 
      ;%7am.temp.source when we use chr(34) as our common character. 
      aline @7am Start of .2. $+ %7am.temp.source $+ . section 
    } 
    IF ((</ul> == $1-) && (%7am.temp.source != $null)) { 
      ;If you look at the source of the page we are downloading (http://www.7am.com/nzwires/) 
      ;you should notice that once a section has ended, a single line with just  is sent. 
      ;We can use this to tell the person who typed  /7am that the section has ended. 
      aline @7am End of .2. $+ %7am.temp.source $+ . section 
      UNSET %7am.temp.source 
    } 
    IF (<LI><A?HREF="*>*</A> iswm $1-) { 
      ;Once again if you look at the source of the page that we are downloading, you 
      ;should notice that all the news items are put into the page using a similar format: 
      ;<LI><A?HREF="*>* 
      ;We use this to find each news item, and display it to the window. 
      VAR %7am.temp = $deltok($3,1,62) $4- 
      ;This variable removes the <LI> at the start of the text 
      aline @7am $remove(%7am.temp,</A>,</I>,<I>) ( $+ $deltok($2,1,34) $+ ) 
      ; $remove(%7am.temp,</A>,</I>,<I>)  contains the headline itself, and because this 
      ;is from a HTML page we need to remove any HTML tags such as </A> </I> and <I>. 
      ;$deltok($2,1,34) Contains the URL of the page where the news came from. 
    } 
  } 
} 
on *:SOCKCLOSE:7am:{
  aline @7am -
  aline @7am End Of News from http://www.7am.com/nzwires/
}
Now if we did not use the tokenize command we would have had to use $gettok(%7am,1,32) instead of $1, $gettok(%7am,2,32) for $2 etc... It saves space in the file, and makes the file easier to read (imho).


5. Practical Example: Swear Kicker  Back to Top

As the title says, this example is of a swear kicker. It allows you to set what channels you wish to run the swear kicker on, and what words to kick on. Both the channel list and the swear word list are edited using tokens. Simply paste the below code into your remotes:

menu channel { 
  Swear Kicker 
  . $iif($istok(%swear.chan,#,44),Remove [ # ] ,Add [ # ] ) : { 
    ;This $iif checks to see if the current channel is in the %swear.chans.
    ;Each chan is seperated by a comma if the chan is inside the list,
    ;then the popup should display "Remove #channel", 
    ;otherwise it should return "Add #channel" 
    SET %swear.chan $iif($istok(%swear.chan,#,44),$remtok(%swear.chan,#,1,46),$addtok(%swear.chan,#,44)) 
    ;The $iif here is based on the same concept as the first $iif.
    ;If the channel is in the variable %swear.chans, then 
    ;we need to use $remtok, if the channel isn't, we need to use $addtok.
    ;Don't worry too much if you can't understand this. 
    ;The important thing is what the $istok, $remtok and $addtok do for the script. 
  } 
  . $iif($numtok(%swear.chan,44) > 0,Currently working on) %swear.chan : { 
    echo -a *** Swear Kick Channels: %swear.chan 
  } 
  .- 
  .Add Swearword: { 
    VAR %swear.word.temp = $$?="Enter Swear Word To Add (Wildcards * and ? accepted)" 
    ;Because we use the $wildtok identifier when searching text for swear words, wildcards such as * and ? are 
    ;accepted. Because we use $addtok, if we enter a swearword which is already in the list, it won't be re-added. 
    ;This saves us having to code a check. It also saves us time when we check peoples text for swearwords. 
    SET %swear.word $addtok(%swear.word,%swear.word.temp,44) 
  } 
  .Del Swearword: { 
    VAR %swear.word.temp = $$?="Enter Swear Word To Remove" 
    ;This just removes ($remtok) the swear word that you entered. 
    SET %swear.word $remtok(%swear.word,%swear.word.temp,1,44) 
  } 
  . $iif($numtok(%swear.word,44) > 0,Currently kicking on) %swear.word : { 
    echo -a *** Swear Kick Words: %swear.word 
  } 
} 

on @*:TEXT:*:#: { 
  IF ($istok(%swear.chan,#,44)) { 
    ;If the channel is a token in the %swear.chan variable, then we can proceed. 
    VAR %swear.temp.total.words = $numtok(%swear.word,44) 
    ;Sets a variable to the total number of tokens (the swear words) in the %swear.word variable 
    VAR %swear.temp.current.word = 0 
    ;Sets a variable to keep track of the token that we want to check. 
    :start 
    INC %swear.temp.current.word 
    ;Increases the variable that keeps track of the token we want to check. 
    ;When we increase it, we are saying we 
    ;want to look at the next token, i.e. the next swear word in our swear word list. 
    IF ($wildtok($1-,$gettok(%swear.word,%swear.temp.current.word,44),0,32)) { 
      ;$gettok(%swear.word,%swear.temp.current.word,44) 
      ;returns the swear word we are currently checking for 
      ;$wildtok makes sure that the token is in what the person was saying.
      ;I use $wildtok and not the isin operator 
      ;because it is in my opinion more accurate. 
      :We have the common problem with swear words: "Ass" as a swear word 
      ;would normally mean people who say words like "assassins" would be kicked.
      ;But because $wildtok checks to see 
      ;that just that token is in the text, it would recognise 
      ;that "assassins" is not "ass", thus the person would not be kicked. 
      ;However $wildtok supports wildcards. So we could add a swearword "hell*".
      :This would kick on hell and all its 
      ;variations (e.g. hellish, hellfire, etc...). 
      ban -u60 # $nick 3 
      kick # $nick Do not swear! 
      ;But if the swear word is detected in the persons text, 
      ; we need to *cough* take care of them. 
    } 
    IF (%swear.temp.current.word < %swear.temp.total.words) { GOTO start }
    ;If we haven't checked what the person said against every swear word on our list, 
    ;we need to go back to start and 
    ;check the next swear word. If we have checked them all, we are done! 
  } 
}


6. Practical Example: Userlist Viewer  Back to Top

This example lets you list the contents of your users section (in the specified ini file).
Simply paste the following code into your remotes:
alias userlist.entries { 
  ;usage: /userlist.entries  
  VAR %userlist.entries.total = $lines($1) 
  ;Set a variable to the total number of lines in the file 
  VAR %userlist.entries.current = 0 
  ;Sets a variable to keep track of the line number that we are reading 
  window -l @Userlist-Viewer 
  aline 2 @Userlist-Viewer Userlist: 
  aline 2 @Userlist-Viewer - 
  ;Creates a @window 
  :start 
  INC %userlist.entries.current 
  ;Increases the line number of the file that we will be reading from 
  VAR %userlist.entires.temp = $read -l $+ %userlist.entries.current $1 
  ;Sets a variable to the text from the line in the file 
  IF (%userlist.entires.temp == $null) { UNSET %userlist.entries.area } 
  ;If nothing was in that line that we just read, 
  ;then we can assume that the section just stopped. 
  ;We then unset a variable so the script won't try process the text, 
  ;because its not relevant. 
  IF (%userlist.entries.area == users) { 
    ;If we are still in the relevant section ([users]) 
    ;then we should process its text. 
    ;The format for a user entry is n[number]=level:address 
    ;Note how the = separates the two. The ascii value for a = is 61 ($asc(=)) 
  aline @Userlist-Viewer $deltok(%userlist.entires.temp,1,61) 
    ;We add the level:address to the @window. 
    ;We don't add the n[number]= part 
    ;because I feel that it is irrelevant. 
  } 
  IF (%userlist.entires.temp == [users]) { VAR %userlist.entries.area = users }
  ;If the line that we just read was [users], 
  ;then its safe to assume we have just started reading 
  ;the [users] section of the file. 
  ;So we set a variable to say this. 
  ;In future the script will check if this variable is set. 
  ;If it is set, then the script knows 
  ;that we are in the [users] section, 
  ;and so we should process the text further. 
  IF (%userlist.entries.current < %userlist.entries.total) { GOTO start } 
  ;If we haven't finished going through the entire file, 
  ;we need to go back to start to check the next line. 
}


7. Practical Example: Dice Roller  Back to Top

This example does the very popular channel game of rolling dice. By itself it may not seem like such a great thing, but you can find this concept applied to many RPG scripts or dare I say it, Pokemon scripts. Simply paste the following code into your remotes:
on *:TEXT:!roll *!*:#: { 
  ;To use: someone in the channel other than yourself types: 
  ;!roll number.of.dice!number.of.throws 
  ;e.g. !roll 2!4 will roll 2 dice 4 times. 
  VAR %die.count = 0 
  ;Sets a variable to 0, this variable will keep track of the numbers
  ;that the dice turn up. 
  msg $chan Rolling $gettok($2,1,33) dice $gettok($2,2,33) times. 
  ;Messages the chan saying that it has commenced the roll. 
  VAR %total.dice.to.roll = $gettok($2,1,33) 
  ;Sets a variable to the number of dice you need to use in each throw 
  VAR %how.many.times.to.throw = $gettok($2,2,33) 
  ;Sets a variable to the number of times we need to throw the dice 
  IF ((%total.dice.to.roll !isnum) || (%how.many.times.to.throw !isnum)) { 
    msg $chan Hey $nick! You need to use numbers!
    halt
  } 
  ;The above if statement checks that both variables are numbers,
  ;so we don't get any errors if we continue. 
  VAR %current.throw = 0 
  ;Sets a variable to keep track of the current throw 
  :start.throw 
  INC %current.throw 
  ;The INC command increases the current throw number 
  VAR %current.die = 0 
  ;Sets a variable to the current die number that we are going to roll 
  :start.roll 
  INC %current.die 
  ;Increases the current die number - so we are rolling the next die. 
  INC %die.count $rand(1,6) 
  ;Increases the variable that keeps track of the numbers that the die turn up by 
  ;whatever number turned up on this dice. $rand(1,6) simulates rolling a 6 sided die. 
  IF (%current.die < %total.dice.to.roll) { GOTO start.roll }
  ;If we havn't rolled all the dice for this throw, we need to keep on rolling 
  IF (%current.throw < %how.many.times.to.throw) { GOTO start.throw }
  ;If we havn't done all the throws we were asked to do, we need to keep on throwing. 
  msg $chan Result for $2 $+ : %die.count 
  ;Messages the channel with the result for the throw. 
}