ENHANCED: library(crypto): Retain a compact internal representation of the ciphertext...

This allows efficient decryption also of very large files and long
strings.

It is usually highly advisable to stick to common and portable library
predicates such as append/3. However, since append/3 does not yet
recognize this opportunity for improvement, I apply it manually in
this case, so that also very large files can be efficiently decrypted
using the compact internal string representation.

Without this change, decrypting a 1 GB file takes 48 GB of RAM,
whereas with this change, it only takes 2 GB (one for the string, one
for its copy with the appended tag).

diff --git a/src/lib/crypto.pl b/src/lib/crypto.pl
index efc478ae70201239b226b95351082bdd1f67f625..63696b5c24ca7cb2f234ed9516248f19b6924d4a 100644 (file)
--- a/src/lib/crypto.pl
+++ b/src/lib/crypto.pl
@@ -1,5 +1,5 @@
 /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-   Written 2020, 2021 by Markus Triska ([email protected])
+   Written 2020, 2021, 2022 by Markus Triska ([email protected])
    Part of Scryer Prolog.
 
    Predicates for cryptographic applications.
@@ -46,6 +46,7 @@
 :- use_module(library(format)).
 :- use_module(library(charsio)).
 :- use_module(library(si)).
+:- use_module(library(iso_ext), [partial_string/3]).
 
 /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    hex_bytes(?Hex, ?Bytes) is det.
@@ -594,7 +595,9 @@ crypto_data_decrypt(CipherText0, Algorithm, Key, IV, PlainText, Options) :-
         member(Encoding, [utf8,octet]),
         encoding_chars(octet, CipherText0, CipherText1),
         maplist(char_code, TagChars, Tag),
-        append(CipherText1, TagChars, CipherText),
+        % we append the tag very efficiently, retaining a compact
+        % internal string representation of the ciphertext
+        partial_string(CipherText1, CipherText, TagChars),
         (   Algorithm = 'chacha20-poly1305' -> true
         ;   domain_error('chacha20-poly1305', Algorithm, crypto_data_decrypt/6)
         ),