Build a Bitcoin Mempool observer

The Bitcoin mempool is an important part of the technology. This is the first of two articles on how to build a mempool observer. The backend, which serves as a server to our client application, will be covered in this post. Let's start with a basic understanding of what a mempool observer does. What exactly is a bitcoin mempool?

A mempool observer is a program that lets you browse or view the mempool's various activities (in this case, bitcoin transactions). Before a block can be accepted, all valid bitcoin transactions must first enter a waiting area. The mempool is the name for this waiting area. All valid bitcoin transactions are supposed to pass through the mempool yet not all valid transactions are guaranteed to pass into the the mempool. This is because if the mempool is full, then only new (valid) transactions whose fees are high enough to bump out a lower fee transaction will be added to the mempool. Miners choose transactions to include in a block from the mempool. When the mempool is huge, it signals that there is a lot of traffic on the network, which means that transaction confirmation times are longer and transaction fees are greater.

Let's set up our code base now that we have a fundamental understanding of the mempool. Two folders will be required: one for our server (backend) and the other for our client app. However, there are several prerequisites that must be met before you can begin working on this project:

• Bitcoin core is already installed on your computer and

• Rust 1.5 or later.

We'll also be using Signet for testing purposes.

Next, open your terminal and create a mempool-analyzer directory.

mkdir mempool-analyzer

cd in to the directory and create a directory named server (name it whatever you want) using the cargo command,

cargo new server

This will provide a rust template code base for us to work with. Following that, we'll require the following libraries or dependencies:

• Rocket - for creating endpoints that will be broadcast on an IP address for our client app to fetch data from.
• Bitcoin-rpc - to allow us communicate with bitcoin core from our Rust app
• Serde - to serialize or deserialize our data to a nice format
• Lazy static (optional) - to declare global constants that are executed at run time in rust with ease.

[dependencies]
rocket = {version = "0.5.0-rc.1", features = ["json"]}
serde = {version = "1.0.117", features = ["derive"]}
serde_json="1.0.59"
bitcoincore-rpc = { path = "/Users/{username}/{path to cloned repo}/rust-bitcoincore-rpc/client" }
bitcoincore-rpc-json = "0.14.0"

Open the server folder in your code editor and add these dependencies above to your cargo.toml file, then run the program in your terminal.

cargo run

Note that the bitcoincore-rpc used here is a bit different from the official release. This is because this version is a cloned repository of the master branch. You can integrate this in your computer by first cloning the repo, git checkout to the mempool branch and then pulling this PR from github.com/rust-bitcoin/rust-bitcoincore-rp... Then do a git rebase to merge all changes to the master branch

If all went well, you should see hello world! printed to your console. Let's start by adding some macros to the top of our main.rs file. These will assist in the addition of functional code from our dependencies.

#[macro_use]
extern crate lazy_static;
#[macro_use]
extern crate rocket;
extern crate bitcoincore_rpc;

We also bring in some standard libraries from our dependencies, which I'll go over later in this article.

use bitcoincore_rpc::{
    bitcoin::{hash_types::Txid, hashes::sha256d::Hash},
    json::{self, GetMempoolEntryResult},
    Auth, Client, RpcApi,
};

use rocket::{
    serde::json::Json,
    Build, Rocket,
};
use serde::{Deserialize, Serialize};
use std::{collections::HashMap, vec};

Next, we use the lazy static crate to declare a static variable, which tells Rust to initialize this expression on first access and then treat it as if it were a regular static variable. To break down the expression, we define a variable named "RPC" of type Client (from the bitcoincore_rpc standard library we imported earlier) and we create a new instance of the Client specifying the port on which our bitcoin core is running (localhost:38332 for signet, the port can be different depending on your bitcoin.conf settings) and the Auth, which should match the exact same user settings we defined in our bitcoin.conf file. The method to string() is self-explanatory. The unwrap method catches any errors and panic the program, otherwise it returns the result.

lazy_static! {
    static ref RPC: Client = Client::new(
        &"http://localhost:38332".to_string(),
        Auth::UserPass("lnd".to_string(), "lightning".to_string()),
    )
    .unwrap();
}

Modify your bitcoin.conf file to use the same username and password as is shown above or pass in your username and password for your node in place of lnd and lightning. Also, make sure you have a wallet loaded in bitcoin core. You can load your wallet using the rpc command bitcoin-cli loadwallet <wallet name>

Next, we create two structs:

• MemData: This will organize and store the data we get from the bitcoin core mempool. We establish a data type named txn ids with a HashMap that will be of two types - transaction id and its associated data in JSON format, given that our mempool data is made up of transaction ids and their respective data. Because our result is returned as key-value pairs, we use a HashMap.
• TxnInput: This is just for structuring our input into a type string.

The Debug macro helps detects potential errors in our code and Serialize macro returns a serialized data usually used to persist the object.

#[derive(Debug, Serialize)]
struct MemData {
    txn_ids: HashMap<Txid, json::GetMempoolEntryResult>,
}

#[derive(Debug, Serialize, Deserialize)]
struct TxnInput {
    txn_id: String,
}

Now, we define a function, get_mempool_txns, that gets our mempool data from bitcoin core. We first initialize a variable, raw_mem_pool by assigning it to the result of the method get_raw_mempool_verbose() associated with our RPC client static we defined earlier. If we have the data, the match statement checks for it and returns it in the MemData struct. It returns an error if this isn't the case.

The verbose added to the get_raw_mempool() ensures that we not only get the transaction ids but the corresponding transaction data

fn get_mempool_txns() -> Result<MemData, String> {
    let raw_mem_pool: Result<HashMap<Txid, json::GetMempoolEntryResult>, bitcoincore_rpc::Error> =
        RPC.get_raw_mempool_verbose();

    match raw_mem_pool {
        Ok(data) => {
            println!("mempool data: {:#?}", (data));
            Ok(MemData { txn_ids: data })
        }
        Err(_) => return Err("No data found! {}".into()),
    }
}

It's time to make our findings public so that our client application may use them. We make use of the rocket crate by adding the get macro followed by our endpoint name. As a result, fetch mempool data() returns a JSON representation of our MemData. It can do this because we used the get mempool txns() method, which retrieves the mempool data and assigns it to the variable datum.

#[get("/mempool-data")]
fn fetch_mempool_data() -> Json<MemData> {
    // Send "All" mempool data to endpoint @localhost:8000/mempool-data
    let datum: MemData = get_mempool_txns().unwrap();
    Json(datum)
}

Now lets test our code by adding this block of code to it.

#[launch]
fn rocket() -> Rocket<Build> {
    //computes all routes for launch to server
    rocket::build().mount(
        "/",
        routes![fetch_mempool_data],
    )
}

Then run the program using the cargo run command. This should open a port at localhost:8000 which we can query for our results. You should also see the data printed in the console or terminal.

Send a get request to localhost:8000/mempool-data in Postman to test the API, which will return the data in JSON format.

Let's finish up our program by adding the final feature or endpoint: retrieving a single mempool transaction's data by its transaction id. This is pretty straight-forward, we first add a post macro from rocket, name the endpoint txn-data, define the data key we expect to recieve as txn_id and ensure that the format its being sent is a JSON.

You have to include application/json in your headers when making a post request due to the format = "application/json

The fetch_txn_data function recieves the txn_id as an input which is converted to a Json struct of TxnInput which we defined earlier and it returns a JSON result of the transaction data. In the function, we declare a variable "id" of type string and make a clone of the input recieved from our endpoint. Since our transaction id is expected to be a type of Txid, we create a hash of the id variable and converts it into a type that is readable by our get_mempool_entry function using the Txid::from_hash method.

#[post("/txn-data", data = "<txn_id>", format = "application/json")]
fn fetch_txn_data(txn_id: Json<TxnInput>) -> Json<json::GetMempoolEntryResult> {
    let id: String = txn_id.txn_id.clone();
    let txnid_hash: Hash = id.parse().unwrap();
    let tx_id: Txid = Txid::from_hash(txnid_hash);
    let data: GetMempoolEntryResult = RPC.get_mempool_entry(&tx_id).unwrap();
    Json(data)
}

Finally, let's modify our launch code to reflect the new endpoint by changing it to this block of code.

#[launch]
fn rocket() -> Rocket<Build> {
    //computes all routes for launch to server
    rocket::build().mount(
        "/",
        routes![fetch_mempool_data, fetch_txn_data],
    )
}

To test the new endpoint, using postman make a post request to localhost:8000/txn-data and pass one of the transaction id gotten from our previous endpoint in to the body of your request in JSON format. You should get a response showing the transaction data of the id.

Now, we have two working endpoints - one to fetch all mempool transactions and the other to get transaction data for a single transaction. In the next article for this series, we'll cover how to display our data in a nice format in our client application.