Tokenizing Strings with Ruby

This tutorial will walk you through the process of tokenizing strings using Ruby. Tokenizing is the process of splitting a string into smaller parts, or tokens, based on a specified delimiter.

Beginner30 minutes

Step 1: Introduction to Strings in Ruby

In Ruby, a string is simply a sequence of characters. They can be created using either single quotes (' ') or double quotes (" ").

"This is a string in Ruby."

Practice Exercise

Create a string with your name and print it.

Show Solution

name = "John Doe"
puts name

Step 2: Understanding String Tokenization

String tokenization is the process of breaking down a string into smaller parts, or 'tokens', based on a specified delimiter. In Ruby, we use the `split` method for this.

Practice Exercise

Given a sentence, tokenize it into words. For instance, 'Ruby is fun' should be tokenized into ['Ruby', 'is', 'fun'].

Show Solution

sentence = 'Ruby is fun'
tokens = sentence.split(' ')
puts tokens

Step 3: Using Different Delimiters

You can use any character as a delimiter in the `split` method. For instance, if you want to tokenize a string into sentences, you could use a period ('.') as your delimiter.

text = "This is sentence one. This is sentence two."
sentences = text.split('.')

Practice Exercise

Given a string of data separated by commas, tokenize it into an array of data points. For instance, '1,2,3,4,5' should be tokenized into ['1', '2', '3', '4', '5'].

Show Solution

data = '1,2,3,4,5'
data_points = data.split(',')
puts data_points

Step 4: Removing Extra Whitespace

After tokenizing a string, you may be left with extra whitespace. You can remove this by chaining the `strip` method after `split`.

text = "This is sentence one. This is sentence two."
sentences = text.split('.').map(&:strip)

Practice Exercise

Given a string of data separated by commas, tokenize it into an array of data points and remove any leading or trailing whitespace. For example, ' 1, 2, 3, 4, 5 ' should be tokenized into ['1', '2', '3', '4', '5'].

Show Solution

data = ' 1, 2, 3, 4, 5 '
data_points = data.split(',').map(&:strip)
puts data_points

Step 5: Conclusion

Tokenizing strings in Ruby is a powerful tool that can be used to parse and manipulate data. By using different delimiters and the `strip` method, you can customize your tokenization to fit any need.

Practice Exercise

Given a paragraph of text, tokenize it into sentences, then tokenize each sentence into words. Remove any extra whitespace.

Show Solution

paragraph = "This is sentence one. This is sentence two."
sentences = paragraph.split('.').map(&:strip)
words = sentences.map { |sentence| sentence.split(' ') }
puts words