Crate bstr

Expand description

A byte string library.

Byte strings are just like standard Unicode strings with one very important difference: byte strings are only conventionally UTF-8 while Rust’s standard Unicode strings are guaranteed to be valid UTF-8. The primary motivation for byte strings is for handling arbitrary bytes that are mostly UTF-8.

Overview

This crate provides two important traits that provide string oriented methods on &[u8] and Vec<u8> types:

ByteSlice extends the [u8] type with additional string oriented methods.
ByteVec extends the Vec<u8> type with additional string oriented methods.

Additionally, this crate provides two concrete byte string types that deref to [u8] and Vec<u8>. These are useful for storing byte string types, and come with convenient std::fmt::Debug implementations:

BStr is a byte string slice, analogous to str.
BString is an owned growable byte string buffer, analogous to String.

Additionally, the free function B serves as a convenient short hand for writing byte string literals.

Quick examples

Byte strings build on the existing APIs for Vec<u8> and &[u8], with additional string oriented methods. Operations such as iterating over graphemes, searching for substrings, replacing substrings, trimming and case conversion are examples of things not provided on the standard library &[u8] APIs but are provided by this crate. For example, this code iterates over all of occurrences of a subtring:

use bstr::ByteSlice;

let s = b"foo bar foo foo quux foo";

let mut matches = vec![];
for start in s.find_iter("foo") {
    matches.push(start);
}
assert_eq!(matches, [0, 8, 12, 21]);

Here’s another example showing how to do a search and replace (and also showing use of the B function):

use bstr::{B, ByteSlice};

let old = B("foo ☃☃☃ foo foo quux foo");
let new = old.replace("foo", "hello");
assert_eq!(new, B("hello ☃☃☃ hello hello quux hello"));

And here’s an example that shows case conversion, even in the presence of invalid UTF-8:

use bstr::{ByteSlice, ByteVec};

let mut lower = Vec::from("hello β");
lower[0] = b'\xFF';
// lowercase β is uppercased to Β
assert_eq!(lower.to_uppercase(), b"\xFFELLO \xCE\x92");

Convenient debug representation

When working with byte strings, it is often useful to be able to print them as if they were byte strings and not sequences of integers. While this crate cannot affect the std::fmt::Debug implementations for [u8] and Vec<u8>, this crate does provide the BStr and BString types which have convenient std::fmt::Debug implementations.

For example, this

use bstr::ByteSlice;

let mut bytes = Vec::from("hello β");
bytes[0] = b'\xFF';

println!("{:?}", bytes.as_bstr());

will output "\xFFello β".

This example works because the ByteSlice::as_bstr method converts any &[u8] to a &BStr.

When should I use byte strings?

This library reflects my hypothesis that UTF-8 by convention is a better trade off in some circumstances than guaranteed UTF-8. It’s possible, perhaps even likely, that this is a niche concern for folks working closely with core text primitives.

The first time this idea hit me was in the implementation of Rust’s regex engine. In particular, very little of the internal implementation cares at all about searching valid UTF-8 encoded strings. Indeed, internally, the implementation converts &str from the API to &[u8] fairly quickly and just deals with raw bytes. UTF-8 match boundaries are then guaranteed by the finite state machine itself rather than any specific string type. This makes it possible to not only run regexes on &str values, but also on &[u8] values.

Why would you ever want to run a regex on a &[u8] though? Well, &[u8] is the fundamental way at which one reads data from all sorts of streams, via the standard library’s Read trait. In particular, there is no platform independent way to determine whether what you’re reading from is some binary file or a human readable text file. Therefore, if you’re writing a program to search files, you probably need to deal with &[u8] directly unless you’re okay with first converting it to a &str and dropping any bytes that aren’t valid UTF-8. (Or otherwise determine the encoding—which is often impractical—and perform a transcoding step.) Often, the simplest and most robust way to approach this is to simply treat the contents of a file as if it were mostly valid UTF-8 and pass through invalid UTF-8 untouched. This may not be the most correct approach though!

One case in particular exacerbates these issues, and that’s memory mapping a file. When you memory map a file, that file may be gigabytes big, but all you get is a &[u8]. Converting that to a &str all in one go is generally not a good idea because of the costs associated with doing so, and also because it generally causes one to do two passes over the data instead of one, which is quite undesirable. It is of course usually possible to do it an incremental way by only parsing chunks at a time, but this is often complex to do or impractical. For example, many regex engines only accept one contiguous sequence of bytes at a time with no way to perform incremental matching.

In summary, conventional UTF-8 byte strings provided by this library are definitely useful in some limited circumstances, but how useful they are more broadly isn’t clear yet.

`bstr` in public APIs

Since this library is not yet 1.0, you should not use it in the public API of your crates until it hits 1.0 (unless you’re OK with with tracking breaking releases of bstr). It is expected that bstr 1.0 will be released before 2022.

In general, it should be possible to avoid putting anything in this crate into your public APIs. Namely, you should never need to use the ByteSlice or ByteVec traits as bounds on public APIs, since their only purpose is to extend the methods on the concrete types [u8] and Vec<u8>, respectively. Similarly, it should not be necessary to put either the BStr or BString types into public APIs. If you want to use them internally, then they can be converted to/from [u8]/Vec<u8> as needed.

Differences with standard strings

The primary difference between [u8] and str is that the former is conventionally UTF-8 while the latter is guaranteed to be UTF-8. The phrase “conventionally UTF-8” means that a [u8] may contain bytes that do not form a valid UTF-8 sequence, but operations defined on the type in this crate are generally most useful on valid UTF-8 sequences. For example, iterating over Unicode codepoints or grapheme clusters is an operation that is only defined on valid UTF-8. Therefore, when invalid UTF-8 is encountered, the Unicode replacement codepoint is substituted. Thus, a byte string that is not UTF-8 at all is of limited utility when using these crate.

However, not all operations on byte strings are specifically Unicode aware. For example, substring search has no specific Unicode semantics ascribed to it. It works just as well for byte strings that are completely valid UTF-8 as for byte strings that contain no valid UTF-8 at all. Similarly for replacements and various other operations that do not need any Unicode specific tailoring.

Aside from the difference in how UTF-8 is handled, the APIs between [u8] and str (and Vec<u8> and String) are intentionally very similar, including maintaining the same behavior for corner cases in things like substring splitting. There are, however, some differences:

Substring search is not done with matches, but instead, find_iter. In general, this crate does not define any generic Pattern infrastructure, and instead prefers adding new methods for different argument types. For example, matches can search by a char or a &str, where as find_iter can only search by a byte string. find_char can be used for searching by a char.
Since SliceConcatExt in the standard library is unstable, it is not possible to reuse that to implement join and concat methods. Instead, join and concat are provided as free functions that perform a similar task.
This library bundles in a few more Unicode operations, such as grapheme, word and sentence iterators. More operations, such as normalization and case folding, may be provided in the future.
Some String/str APIs will panic if a particular index was not on a valid UTF-8 code unit sequence boundary. Conversely, no such checking is performed in this crate, as is consistent with treating byte strings as a sequence of bytes. This means callers are responsible for maintaining a UTF-8 invariant if that’s important.
Some routines provided by this crate, such as starts_with_str, have a _str suffix to differentiate them from similar routines already defined on the [u8] type. The difference is that starts_with requires its parameter to be a &[u8], where as starts_with_str permits its parameter to by anything that implements AsRef<[u8]>, which is more flexible. This means you can write bytes.starts_with_str("☃") instead of bytes.starts_with("☃".as_bytes()).

Otherwise, you should find most of the APIs between this crate and the standard library string APIs to be very similar, if not identical.

Handling of invalid UTF-8

Since byte strings are only conventionally UTF-8, there is no guarantee that byte strings contain valid UTF-8. Indeed, it is perfectly legal for a byte string to contain arbitrary bytes. However, since this library defines a string type, it provides many operations specified by Unicode. These operations are typically only defined over codepoints, and thus have no real meaning on bytes that are invalid UTF-8 because they do not map to a particular codepoint.

For this reason, whenever operations defined only on codepoints are used, this library will automatically convert invalid UTF-8 to the Unicode replacement codepoint, U+FFFD, which looks like this: �. For example, an iterator over codepoints will yield a Unicode replacement codepoint whenever it comes across bytes that are not valid UTF-8:

use bstr::ByteSlice;

let bs = b"a\xFF\xFFz";
let chars: Vec<char> = bs.chars().collect();
assert_eq!(vec!['a', '\u{FFFD}', '\u{FFFD}', 'z'], chars);

There are a few ways in which invalid bytes can be substituted with a Unicode replacement codepoint. One way, not used by this crate, is to replace every individual invalid byte with a single replacement codepoint. In contrast, the approach this crate uses is called the “substitution of maximal subparts,” as specified by the Unicode Standard (Chapter 3, Section 9). (This approach is also used by W3C’s Encoding Standard.) In this strategy, a replacement codepoint is inserted whenever a byte is found that cannot possibly lead to a valid UTF-8 code unit sequence. If there were previous bytes that represented a prefix of a well-formed UTF-8 code unit sequence, then all of those bytes (up to 3) are substituted with a single replacement codepoint. For example:

use bstr::ByteSlice;

let bs = b"a\xF0\x9F\x87z";
let chars: Vec<char> = bs.chars().collect();
// The bytes \xF0\x9F\x87 could lead to a valid UTF-8 sequence, but 3 of them
// on their own are invalid. Only one replacement codepoint is substituted,
// which demonstrates the "substitution of maximal subparts" strategy.
assert_eq!(vec!['a', '\u{FFFD}', 'z'], chars);

If you do need to access the raw bytes for some reason in an iterator like Chars, then you should use the iterator’s “indices” variant, which gives the byte offsets containing the invalid UTF-8 bytes that were substituted with the replacement codepoint. For example:

use bstr::{B, ByteSlice};

let bs = b"a\xE2\x98z";
let chars: Vec<(usize, usize, char)> = bs.char_indices().collect();
// Even though the replacement codepoint is encoded as 3 bytes itself, the
// byte range given here is only two bytes, corresponding to the original
// raw bytes.
assert_eq!(vec![(0, 1, 'a'), (1, 3, '\u{FFFD}'), (3, 4, 'z')], chars);

// Thus, getting the original raw bytes is as simple as slicing the original
// byte string:
let chars: Vec<&[u8]> = bs.char_indices().map(|(s, e, _)| &bs[s..e]).collect();
assert_eq!(vec![B("a"), B(b"\xE2\x98"), B("z")], chars);

File paths and OS strings

One of the premiere features of Rust’s standard library is how it handles file paths. In particular, it makes it very hard to write incorrect code while simultaneously providing a correct cross platform abstraction for manipulating file paths. The key challenge that one faces with file paths across platforms is derived from the following observations:

On most Unix-like systems, file paths are an arbitrary sequence of bytes.
On Windows, file paths are an arbitrary sequence of 16-bit integers.

(In both cases, certain sequences aren’t allowed. For example a NUL byte is not allowed in either case. But we can ignore this for the purposes of this section.)

Byte strings, like the ones provided in this crate, line up really well with file paths on Unix like systems, which are themselves just arbitrary sequences of bytes. It turns out that if you treat them as “mostly UTF-8,” then things work out pretty well. On the contrary, byte strings don’t really work that well on Windows because it’s not possible to correctly roundtrip file paths between 16-bit integers and something that looks like UTF-8 without explicitly defining an encoding to do this for you, which is anathema to byte strings, which are just bytes.

Rust’s standard library elegantly solves this problem by specifying an internal encoding for file paths that’s only used on Windows called WTF-8. Its key properties are that they permit losslessly roundtripping file paths on Windows by extending UTF-8 to support an encoding of surrogate codepoints, while simultaneously supporting zero-cost conversion from Rust’s Unicode strings to file paths. (Since UTF-8 is a proper subset of WTF-8.)

The fundamental point at which the above strategy fails is when you want to treat file paths as things that look like strings in a zero cost way. In most cases, this is actually the wrong thing to do, but some cases call for it, for example, glob or regex matching on file paths. This is because WTF-8 is treated as an internal implementation detail, and there is no way to access those bytes via a public API. Therefore, such consumers are limited in what they can do:

One could re-implement WTF-8 and re-encode file paths on Windows to WTF-8 by accessing their underlying 16-bit integer representation. Unfortunately, this isn’t zero cost (it introduces a second WTF-8 decoding step) and it’s not clear this is a good thing to do, since WTF-8 should ideally remain an internal implementation detail.
One could instead declare that they will not handle paths on Windows that are not valid UTF-16, and return an error when one is encountered.
Like (2), but instead of returning an error, lossily decode the file path on Windows that isn’t valid UTF-16 into UTF-16 by replacing invalid bytes with the Unicode replacement codepoint.

While this library may provide facilities for (1) in the future, currently, this library only provides facilities for (2) and (3). In particular, a suite of conversion functions are provided that permit converting between byte strings, OS strings and file paths. For owned byte strings, they are:

For byte string slices, they are:

On Unix, all of these conversions are rigorously zero cost, which gives one a way to ergonomically deal with raw file paths exactly as they are using normal string-related functions. On Windows, these conversion routines perform a UTF-8 check and either return an error or lossily decode the file path into valid UTF-8, depending on which function you use. This means that you cannot roundtrip all file paths on Windows correctly using these conversion routines. However, this may be an acceptable downside since such file paths are exceptionally rare. Moreover, roundtripping isn’t always necessary, for example, if all you’re doing is filtering based on file paths.

The reason why using byte strings for this is potentially superior than the standard library’s approach is that a lot of Rust code is already lossily converting file paths to Rust’s Unicode strings, which are required to be valid UTF-8, and thus contain latent bugs on Unix where paths with invalid UTF-8 are not terribly uncommon. If you instead use byte strings, then you’re guaranteed to write correct code for Unix, at the cost of getting a corner case wrong on Windows.