Tokenize a string in C++

This post will discuss how to tokenize a string in C++.

1. Using String Stream

The common solution to tokenize a string in C++ is using std::istringstream, which is a stream class to operate on strings. The following code extract tokens from the stream using the extraction operator and insert them into a container.

#include <iostream>

#include <vector>

#include <string>

#include <sstream>

#include <iterator>

using namespace std;

int main()

{

string str = "Tokenize a string in C++";

vector<string> tokens;

istringstream iss(str);

string s;

while (iss >> s) {

tokens.push_back(s);

}

for (const string &s: tokens) {

cout << s << endl;

}

return 0;

}

Download Run Code

Output:

Tokenize
a
string
in
C++

Here’s another approach to extract tokens from an input string using std::istream_iterator. The std::istream_iterator skip whitespace by default unless disabled with std::noskipws.

#include <iostream>

#include <vector>

#include <string>

#include <sstream>

#include <iterator>

using namespace std;

int main()

{

string str = "Tokenize a string in C++";

istringstream iss(str);

vector<string> tokens;

copy(istream_iterator<string>(iss), istream_iterator<string>(),

back_inserter(tokens));

for (const string &s: tokens) {

cout << s << endl;

}

return 0;

}

Download Run Code

Output:

Tokenize
a
string
in
C++

The code can be further shortened to the following using the initializer list:

#include <iostream>

#include <vector>

#include <string>

#include <sstream>

#include <iterator>

using namespace std;

int main()

{

string str = "Tokenize a string in C++";

istringstream iss(str);

vector<string> tokens = {istream_iterator<string>{iss}, istream_iterator<string>{}};

for (const string &s: tokens) {

cout << s << endl;

}

return 0;

}

Download Run Code

Output:

Tokenize
a
string
in
C++

2. Using regular expressions

Another plausible solution to tokenize a string is using a regular expression, which is the standard for performing a pattern match against a sequence.

#include <iostream>

#include <vector>

#include <string>

#include <regex>

using namespace std;

int main()

{

string str = "Tokenize a string in C++";

regex reg("\\s+");

sregex_token_iterator iter(str.begin(), str.end(), reg, -1);

sregex_token_iterator end;

vector<string> tokens(iter, end);

for (const string &s: tokens) {

cout << s << endl;

}

return 0;

}

Download Run Code

Output:

Tokenize
a
string
in
C++

3. Using Boost library

The Boost C++ library also offers several utility classes for this task. The Boost tokenizer class provides a view of tokens contained in a sequence by interpreting certain characters as separators.

#include <iostream>

#include <string>

#include <boost/tokenizer.hpp>

using namespace std;

using namespace boost;

int main()

{

string str = "Tokenize a string in C++";

char_separator<char> sep(" ");

tokenizer<char_separator<char>> tokens(str, sep);

for (const string &s: tokens) {

cout << s << endl;

}

return 0;

}

Download Code

Output:

Tokenize
a
string
in
C++

That’s all about tokenizing a string in C++.

1. Using String Stream

2. Using regular expressions

3. Using Boost library

Share this: