Search inside messages, across every conversation
🔴 There is no index, and that is a decision rather than an omission. An index derived from decrypted history is plaintext by another name: it would have to be sealed as carefully as the vault, taken by forgetEverything, and kept consistent with the threads — a second copy of everything anybody ever wrote, with its own ways to go wrong. This reads what already exists and throws the plaintext away. ⚠️ It costs more here than on iOS, and the code says so. The iPhone app keeps the whole decrypted history in memory; Android decrypts each thread on demand. Hence a two-character minimum, a bound, a 250ms debounce, and Dispatchers.IO. 🔴 Folding is length-preserving on purpose: the snippet slices the ORIGINAL string at an index found in the folded one, so a fold that dropped characters would cut in the wrong place — and for an accented character, mid-character. There is a test with accents before the match. Accent-insensitive because somebody typing "cafe" means "café", and a search that knows the difference quietly fails in every language with accents — Spanish is next on the localisation list. Refs paulhitt/flamenet-messenger-ios#25 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Paul Hitt
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
5a0a061a34
commit
42b7d1732f
@@ -0,0 +1,146 @@
|
||||
package io.flamenet.e2e
|
||||
|
||||
import org.json.JSONObject
|
||||
|
||||
/**
|
||||
* Finding a word somebody said, across every conversation on this device.
|
||||
*
|
||||
* 🔴 **There is no index, and that is a decision rather than an omission.** The
|
||||
* obvious design builds a search index over the decrypted history — and an
|
||||
* index derived from plaintext is plaintext by another name, so it would have
|
||||
* to be sealed as carefully as the history itself, purged with it, and kept
|
||||
* consistent with it. That is a second copy of everything anybody ever wrote,
|
||||
* with its own ways to go wrong.
|
||||
*
|
||||
* Instead this reads the threads that already exist, through the vault, and
|
||||
* throws the plaintext away when it is done. Nothing new comes to rest, so
|
||||
* there is nothing new to protect.
|
||||
*
|
||||
* ⚠️ **This costs more here than on iOS**, and the difference is worth naming.
|
||||
* The iPhone app keeps the whole decrypted history in memory, so its search is
|
||||
* a scan of what is already there. Android reads and decrypts each thread on
|
||||
* demand, so a search touches the vault once per conversation. That is why this
|
||||
* takes a minimum query length and a bound, and why callers must run it off the
|
||||
* main thread. If it ever becomes too slow, an index is the answer — and then it
|
||||
* must be sealed with the vault and taken by `forgetEverything`, or the app will
|
||||
* have quietly acquired an unencrypted transcript.
|
||||
*
|
||||
* 🔴 **The query itself is never persisted.** What somebody searched for is
|
||||
* often more revealing than what they found.
|
||||
*/
|
||||
object MessageSearch {
|
||||
|
||||
/**
|
||||
* How many hits are built at once.
|
||||
*
|
||||
* ⚠️ A bound, not a preference. A short query matches most of a long
|
||||
* history, and a list of thousands of rows is a spinner that never ends for
|
||||
* a result nobody reads past the first screen of.
|
||||
*/
|
||||
const val LIMIT = 60
|
||||
|
||||
/**
|
||||
* Below this, a search is not run at all.
|
||||
*
|
||||
* ⚠️ Not a UI nicety. One character matches nearly every message, so the
|
||||
* work is maximal and the result is useless — and unlike iOS, that work
|
||||
* means decrypting every thread on the device.
|
||||
*/
|
||||
const val MINIMUM_QUERY = 2
|
||||
|
||||
/** Characters of context kept either side of the match. */
|
||||
const val CONTEXT = 40
|
||||
|
||||
data class Hit(
|
||||
val threadId: String,
|
||||
val text: String,
|
||||
/** The part of the message around the match. */
|
||||
val snippet: String,
|
||||
val at: Long,
|
||||
val mine: Boolean,
|
||||
)
|
||||
|
||||
/**
|
||||
* Search every thread the vault holds.
|
||||
*
|
||||
* ⚠️ Blocking and disk-bound. The caller is responsible for keeping it off
|
||||
* the main thread.
|
||||
*/
|
||||
fun hits(query: String, vault: Vault, limit: Int = LIMIT): List<Hit> {
|
||||
// 🔴 An empty query finds NOTHING, never everything. Returning the whole
|
||||
// history the moment somebody focuses the search box would put every
|
||||
// message they have on one screen — the place most likely to be read
|
||||
// over a shoulder.
|
||||
val needle = query.trim()
|
||||
if (needle.length < MINIMUM_QUERY) return emptyList()
|
||||
|
||||
val found = mutableListOf<Hit>()
|
||||
for (threadId in vault.threads()) {
|
||||
val stored = vault.history(threadId)
|
||||
for (i in 0 until stored.length()) {
|
||||
val m: JSONObject = stored.getJSONObject(i)
|
||||
val text = m.optString("text")
|
||||
if (!matches(text, needle)) continue
|
||||
found += Hit(
|
||||
threadId = threadId,
|
||||
text = text,
|
||||
snippet = snippet(text, needle),
|
||||
at = m.optLong("at", 0L),
|
||||
mine = m.optBoolean("mine", false),
|
||||
)
|
||||
}
|
||||
}
|
||||
// Newest first: what somebody is looking for is far more often recent
|
||||
// than old, and it is the only ordering that needs no explanation.
|
||||
return found.sortedByDescending { it.at }.take(limit)
|
||||
}
|
||||
|
||||
/**
|
||||
* ⚠️ Case- AND accent-insensitive. Somebody typing `cafe` means `café`, and
|
||||
* a search that knows the difference quietly fails for every language with
|
||||
* accents in it — which is most of them, and this app ships in Spanish next.
|
||||
*
|
||||
* ⚠️ A plain substring, never a regular expression. The needle is typed by
|
||||
* a person: a stray bracket is a crash or a hang, and nobody typing into a
|
||||
* search box intended a pattern.
|
||||
*/
|
||||
fun matches(haystack: String, needle: String): Boolean {
|
||||
if (haystack.isEmpty() || needle.isEmpty()) return false
|
||||
return fold(haystack).contains(fold(needle))
|
||||
}
|
||||
|
||||
/**
|
||||
* The part of a message around the match, so a hit says why it is a hit.
|
||||
*
|
||||
* ⚠️ Bounded on both sides. A whole message means one long one pushes every
|
||||
* other result off the screen; only the start means a match near the end
|
||||
* looks like no match at all.
|
||||
*/
|
||||
fun snippet(text: String, needle: String): String {
|
||||
val at = fold(text).indexOf(fold(needle))
|
||||
if (at < 0) return text.take(CONTEXT * 2)
|
||||
|
||||
val start = (at - CONTEXT).coerceAtLeast(0)
|
||||
val end = (at + needle.length + CONTEXT).coerceAtMost(text.length)
|
||||
val piece = text.substring(start, end)
|
||||
return (if (start > 0) "…" else "") + piece + (if (end < text.length) "…" else "")
|
||||
}
|
||||
|
||||
/**
|
||||
* ⚠️ Folding runs on both sides and must be length-preserving, because
|
||||
* [snippet] slices the ORIGINAL string at an index found in the folded one.
|
||||
* A fold that dropped or added characters would cut the snippet in the
|
||||
* wrong place — and for a combining accent it would cut mid-character.
|
||||
*/
|
||||
private fun fold(s: String): String {
|
||||
val normalised = java.text.Normalizer.normalize(s, java.text.Normalizer.Form.NFD)
|
||||
val stripped = buildString(s.length) {
|
||||
for (ch in normalised) {
|
||||
// Combining marks only. Dropping them from the DECOMPOSED form
|
||||
// leaves one character per original character.
|
||||
if (Character.getType(ch) != Character.NON_SPACING_MARK.toInt()) append(ch)
|
||||
}
|
||||
}
|
||||
return stripped.lowercase()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
package io.flamenet.e2e
|
||||
|
||||
import javax.crypto.KeyGenerator
|
||||
import javax.crypto.SecretKey
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertTrue
|
||||
import org.json.JSONObject
|
||||
|
||||
/**
|
||||
* Finding a word somebody said, across every conversation on this device.
|
||||
*
|
||||
* 🔴 **There is no index, and that is the design.** An index derived from
|
||||
* decrypted history is plaintext by another name: it would have to be sealed as
|
||||
* carefully as the vault, taken by `forgetEverything`, and kept consistent with
|
||||
* the threads. This reads what already exists and throws the plaintext away,
|
||||
* so nothing new comes to rest.
|
||||
*/
|
||||
class MessageSearchTest {
|
||||
|
||||
private class MapStorage : Vault.Storage {
|
||||
val records = LinkedHashMap<String, ByteArray>()
|
||||
override fun read(name: String) = records[name]
|
||||
override fun write(name: String, bytes: ByteArray) { records[name] = bytes }
|
||||
override fun delete(name: String) { records.remove(name) }
|
||||
override fun names() = records.keys.toList()
|
||||
}
|
||||
|
||||
private fun key(): SecretKey =
|
||||
KeyGenerator.getInstance("AES").apply { init(256) }.generateKey()
|
||||
|
||||
private fun vaultWith(vararg lines: Triple<String, String, Long>): Vault {
|
||||
val vault = Vault(MapStorage(), key())
|
||||
for ((threadId, text, at) in lines) {
|
||||
vault.append(threadId, JSONObject()
|
||||
.put("text", text).put("mine", false).put("at", at))
|
||||
}
|
||||
return vault
|
||||
}
|
||||
|
||||
private val now = 1_700_000_000_000L
|
||||
|
||||
// ---------------------------------------------------------------- finding
|
||||
|
||||
@Test
|
||||
fun `it finds a message by its words`() {
|
||||
val vault = vaultWith(
|
||||
Triple(Vault.threadForUser(7), "meet at the cafe at six", now - 3000),
|
||||
Triple(Vault.threadForUser(7), "bring the keys", now - 2000),
|
||||
)
|
||||
val hits = MessageSearch.hits("keys", vault)
|
||||
assertEquals(1, hits.size)
|
||||
assertEquals("bring the keys", hits.first().text)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `matching is case insensitive`() {
|
||||
val vault = vaultWith(
|
||||
Triple(Vault.threadForUser(7), "CAFE is closed", now),
|
||||
Triple(Vault.threadForUser(7), "the cafe again", now - 1000),
|
||||
)
|
||||
assertEquals(2, MessageSearch.hits("cafe", vault).size)
|
||||
}
|
||||
|
||||
/**
|
||||
* ⚠️ Somebody typing `cafe` means `café`. A search that knows the
|
||||
* difference quietly fails for every language with accents in it — which is
|
||||
* most of them, and this app ships in Spanish next.
|
||||
*/
|
||||
@Test
|
||||
fun `matching ignores accents`() {
|
||||
val vault = vaultWith(Triple(Vault.threadForUser(7), "meet at the café", now))
|
||||
assertEquals(1, MessageSearch.hits("cafe", vault).size)
|
||||
assertTrue(MessageSearch.matches("meet at the cafe", "café"))
|
||||
}
|
||||
|
||||
/**
|
||||
* ⚠️ A plain substring, never a regular expression. A stray bracket typed
|
||||
* into a search box is a crash or a hang, and nobody meant a pattern.
|
||||
*/
|
||||
@Test
|
||||
fun `the query is not a pattern`() {
|
||||
val vault = vaultWith(
|
||||
Triple(Vault.threadForUser(7), "the answer is (42)", now),
|
||||
Triple(Vault.threadForUser(7), "unrelated", now - 1000),
|
||||
)
|
||||
assertEquals(1, MessageSearch.hits("(42)", vault).size)
|
||||
// `.*` matches everything as a regex and nothing as a substring.
|
||||
assertTrue(MessageSearch.hits(".*", vault).isEmpty())
|
||||
}
|
||||
|
||||
/** It reaches across every conversation, not just the open one. */
|
||||
@Test
|
||||
fun `it searches every thread`() {
|
||||
val vault = vaultWith(
|
||||
Triple(Vault.threadForUser(7), "hello there", now - 3000),
|
||||
Triple(Vault.threadForGroup("g1"), "hello all", now - 2000),
|
||||
Triple(Vault.threadForAddress("ada@other.example"), "hello afar", now - 1000),
|
||||
)
|
||||
assertEquals(3, MessageSearch.hits("hello", vault).size)
|
||||
assertEquals(
|
||||
setOf(Vault.threadForUser(7), Vault.threadForGroup("g1"),
|
||||
Vault.threadForAddress("ada@other.example")),
|
||||
MessageSearch.hits("hello", vault).map { it.threadId }.toSet())
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- what it must not do
|
||||
|
||||
/**
|
||||
* 🔴 A short query finds NOTHING, never everything. Returning the whole
|
||||
* history the moment somebody focuses the search box would put every
|
||||
* message they have on one screen — and here it would decrypt every thread
|
||||
* on the device to do it.
|
||||
*/
|
||||
@Test
|
||||
fun `a query below the minimum finds nothing`() {
|
||||
val vault = vaultWith(Triple(Vault.threadForUser(7), "anything at all", now))
|
||||
for (tooShort in listOf("", " ", "a", " x ")) {
|
||||
assertTrue(MessageSearch.hits(tooShort, vault).isEmpty(),
|
||||
"\"$tooShort\" searched anyway")
|
||||
}
|
||||
}
|
||||
|
||||
/** An empty message must not match everything. */
|
||||
@Test
|
||||
fun `an empty message does not match everything`() {
|
||||
assertFalse(MessageSearch.matches("", "anything"))
|
||||
assertFalse(MessageSearch.matches("anything", ""))
|
||||
}
|
||||
|
||||
/** ⚠️ A bound, not a preference. */
|
||||
@Test
|
||||
fun `results are bounded and keep the newest`() {
|
||||
val vault = Vault(MapStorage(), key())
|
||||
for (i in 1..200) {
|
||||
vault.append(Vault.threadForUser(7), JSONObject()
|
||||
.put("text", "needle $i").put("mine", false).put("at", now - i * 1000L))
|
||||
}
|
||||
val hits = MessageSearch.hits("needle", vault, limit = 3)
|
||||
assertEquals(3, hits.size)
|
||||
assertEquals(listOf("needle 1", "needle 2", "needle 3"), hits.map { it.text })
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- the snippet
|
||||
|
||||
/**
|
||||
* ⚠️ Bounded on both sides. A whole message means one long one pushes every
|
||||
* other result off the screen; only the start means a match near the end
|
||||
* looks like no match at all.
|
||||
*/
|
||||
@Test
|
||||
fun `the snippet is centred on the match`() {
|
||||
val long = "x".repeat(300) + "needle" + "y".repeat(300)
|
||||
val snippet = MessageSearch.snippet(long, "needle")
|
||||
|
||||
assertTrue(snippet.contains("needle"), "the snippet does not show the match")
|
||||
assertTrue(snippet.length < 120, "the snippet is the whole message")
|
||||
assertTrue(snippet.startsWith("…"), "no sign that text was cut from the front")
|
||||
assertTrue(snippet.endsWith("…"), "no sign that text was cut from the end")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a short message is shown whole`() {
|
||||
assertEquals("bring the keys", MessageSearch.snippet("bring the keys", "keys"))
|
||||
}
|
||||
|
||||
/**
|
||||
* 🔴 **Folding has to be length-preserving**, because the snippet slices the
|
||||
* ORIGINAL string at an index found in the folded one. A fold that dropped
|
||||
* or added characters would cut in the wrong place — and for an accented
|
||||
* character, mid-character.
|
||||
*/
|
||||
@Test
|
||||
fun `the snippet is correct when the text has accents before the match`() {
|
||||
val text = "café café café needle here"
|
||||
val snippet = MessageSearch.snippet(text, "needle")
|
||||
assertTrue(snippet.contains("needle"),
|
||||
"accents shifted the slice off the match: $snippet")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user