Looking at Bitcoin transactions

Transactions

In this article I will explain what are the contents of a Bitcoin article and also try to break some common misconceptions about Bitcoin.

Bitcoin is the common name for a currency, a ledger and the protocol by which this ledger is shared and secured between participating nodes. This is an important distinction.
A transaction is a message in the Bitcoin Protocol, that describes a change in the Bitcoin Ledger and may, or may not be used to send Bitcoin Coins somewhere else.
Confused yet?

The bitcoin protocol consists of rules about transaction validations and methods for communicating the transactions with other participants.
The second part can be ignored for now. It is not the focus of this article.
The first part – the validations – must remain the same for all nodes that participate in the same network. If any node has different validation rules, even by a bit, it is likely to isolate itself by forking itself from the network. (There are exceptions to these rules, in the form of soft forks, but for the sake of simplicity, we will ignore this for now.)

A common misconception is that part of the validation is to see if the message is coming from the person who has the coins that are being spent. This is wrong for multiple reasons:
– not all coins are locked in addresses
– the validation of an address (or even the fact that addresses exist) is not something that the validation logic of transactions is concerned with

To understand that, you need to understand what a transaction consists of.

Size Field Description

4 bytes

Version

Specifies which rules this transaction follows

1–9 bytes (VarInt)

Input Counter

How many inputs are included

Variable

Inputs

One or more transaction inputs

1–9 bytes (VarInt)

Output Counter

How many outputs are included

Variable

Outputs

One or more transaction outputs

4 bytes

Locktime

A Unix timestamp or block number

The first things to verify is the version of the rules is supported, and if the transaction is properly formatted (and not some gibberish like “np3r3h8hr3hn293…”).
Once a transaction is parsed as valid object, the Inputs and Outputs are checked.
If all the inputs exist and their consumption is valid, and all the outputs don’t spend more than the total of the inputs, then the transaction is valid.
Lastly, the validators check the Locktime to see if the valid transaction can be spent yet. If so – they include it in a block. If not – it becomes invalid and gets ignored.

But lets look a bit deeper.

inputs and outputs

Simply explained, inputs are the money you use in your transaction, and outputs are the money you produce out of that transaction.

An everyday example would be when Alex wants to buy a new shirt from the store. The shirt costs 34.99 euro. Alex pays 2 bills of 20 euro each.
The inputs are the 2 bills of 20 euro. Each input has it’s own history. They are 2 separate inputs.
The outputs are 2 as well. The first output is the 34.99 that the shop receives. The second output is the 5.01 that Alex gets as change.

Bitcoin Inputs and outputs are very similar to that example.
After the transaction is done, it’s outputs can become inputs for other transactions.

Each input contains a script that specifies the rules for spending it. This script is written in the Bitcoin Script programming language.

It is a common misconception that the Bitcoin ledger and protocol have addresses, balances and keys. This is not true. These things are an abstraction of the bitcoin scripting subset accepted as standard. In other words the bitcoin scripting language (which allows you to do all sorts of cool stuff with your bitcoin) is ignored in most wallet applications, except if it is in one of a few very strict formats. This limitation of capabilities greatly improves the simplicity of writing and using a bitcoin wallet.
It is a good thing.

When somebody wants to consume an input, he simply has to provide an unlocking script, that goes before the script of the input itself.
If the script of the input is written with a meaning like “X… + 3 should equal 5” (X is the unlocking script) then the correct unlocking script would be “2”, because 2 + 3 = 5
In this example one doesn’t need to know any private keys, but simply needed to point to the input he wants to consume and give an unlocking script that matches the locking one.

The simplest example of a standard transaction is the P2PKH (pay to public key hash) transaction.
In it, the script meaning goes like that:

“Xsig, Xpubk… hash160 of Xpubk should equal <expected hash> and Xsig should be a signed message of Xpubk’s private key

Xsig, Xpubk are the expected values from the unlocking script.
Xpubk  proves that we know which public key corresponds to the <expected hash> that is hardcoded in the script
Xsig proves that we posses the private key corresponding to that public key

This is where the misconception about addresses originates from.

Now lets look at the contents of an Input:

Size Field Description

32 bytes

Transaction Hash

Pointer to the transaction containing the UTXO to be spent

4 bytes

Output Index

The index number of the UTXO to be spent; first one is 0

1-9 bytes (VarInt)

Unlocking-Script Size

Unlocking-Script length in bytes, to follow

Variable

Unlocking-Script

A script that fulfills the conditions of the UTXO locking script.

4 bytes

Sequence Number

Currently disabled Tx-replacement feature, set to 0xFFFFFFFF

As you see the input data in the transaction doesn’t actually contain the input itself. It simply points to an Output Index. Since all bitcoin participants agree on the state of the blockchain, the unconsumed transaction outputs (UTXO) would be the same for all of them. An index is enough to find the needed data. The Input data however must provide an unlocking script.

SCRIPT Validation

When is a script valid?

When it executes to the end and the final value is “TRUE”.
If a script ends with a different result, or if it gets interrupted due to wrong usage of it’s methods, it is invalid, and therefore – the input becomes invalid and can not be consumed.

The output scripts also get validated, but they don’t get executed (since their unlocking script is missing). Instead, an output script validation simply checks if what was written is within the syntax of the bitcoin scripting language.

If all inputs and their scripts are valid and the outputs are valid too, and the sum of the consumed coins from the inputs is greater or equal to the sum of the coins in the outputs, then a transaction can be valid.


This is all for now.

This was a simplified article, aiming to explain basic concepts like Script, UTXO, Inputs and Outputs

Go to my article about bitcoin-cli usage and try looking at some transaction data with it.
Experimentation is the most interesting way of learning

You could also read this chapter of the “Mastering Bitcoin Cash” book : https://developer.bitcoin.com/mastering-bitcoin-cash/transactions.html

In the next post I will try to explain the syntax of the Bitcoin Script programming language and how to write your own scripts.

Study hard. Bye!

Getting price history from CoinMarketCap

Since we are working with currency, getting the price history is kinda important.
To follow me through this article, make sure that you:

  • Understand basic Python code
  • Have Python installed on your PC

Easy.
Ok. Let's start.
First we need to define what exactly we are doing.

# Task 1: Get Top 10 currencies by market cap
# Task 2: Get the trading history of the top 10 coins
# Task 3: Make a cool chart of the history

These are the tasks.
Next condition is that we can only use CoinMarketCap.COM
This is the link to it's API documentation : https://coinmarketcap.com/api/#endpoint_listings

As you see (if you actually opened the link above), there are endpoints to get the current state of the coins, but not their history.
(At least that's how it is at the moment of writing this article...)

So... The API can't give us anything useful for our second task.
We need to look for another way...

 

Let's open https://coinmarketcap.com/ and take a look.

If we select Bitcoin, we arrive at this screen:

Screenshot from 2018-06-12 14-13-56

 

From this screenshot it is obvious that CoinMarketCap has the trading history and shows it to the visitors. Let's dig around and see how we could get that history...
Open your browser debugger ('Inspect Element' in Firefox) and go to the network resources.
Refresh the page

Do you see something interesting?
I do.

Screenshot from 2018-06-12 14-18-42

Look at this request - the one I marked on the screenshot.
It is a GET request to https://graphs2.coinmarketcap.com/currencies/bitcoin/

This is exactly what we needed for our task. It is perfect.
It provides us with BTC price, usd price, volume and market cap.
All of them are histories since the beginning of the coin (or to be exact - since the beginning of the coin's measurements)
Bitcoin's history starts at ~100 dollars, which is not the real beginning, but is the only one we have data for.

Also - each historical price is accompanied by a number, like:

[1435624460000, 256.939]

If you find a epoch converter, you will see that 1435624460000 is "Tuesday, June 30, 2015".
So the price on June 30, 2015 was 256 $

Now let's start writing code!

import requests
import json
import matplotlib

These are the requirements we will be needing. Make sure to install matplotlib.

# https://coinmarketcap.com/api/#endpoint_listings
# https://api.coinmarketcap.com/v2/listings/
# https://api.coinmarketcap.com/v2/ticker/

# Task 1: Get Top 10 currencies by market cap

arr = json.loads(
    requests.get("https://api.coinmarketcap.com/v2/ticker/").content
)["data"].values()

top = sorted(
    arr,
    key=lambdai: i["circulating_supply"] * i["quotes"]["USD"]["price"],
    reverse=True
)[:10]

I this first task we use the API /ticker/ endpoint to get the current price information of ALL cryptocurrencies.
Once that is done, we sort them by their market cap, which we calculate using the circulating_supply.
(The order will be different if we use the total_supply.)
The ordering is reversed, because we want biggest first.
Lastly, there is a [:10] which is just a Python trick, telling it to take the elements with indexes between 0 and 10

Now let's try the next part:

# https://graphs2.coinmarketcap.com/currencies/bitcoin/

# Task 2: Get the trading history of the top 10 coins

historyOfTop = []
for i in top:
    history = json.loads(requests.get(
        "https://graphs2.coinmarketcap.com/currencies/" + i["website_slug"] + "/"
    ).content)
    historyOfTop.append(history)

Here we iterate over the top 10 coins.
We make a request for each of them, using the unofficial API we found earlier.
Then we parse the content of the response as JSON and we add it to the collection of trading histories.

Note: This collection of trading histories does not contain any data about which coin it refers to. The only way to know which history is for which coin is to keep the indexes of 'top' and 'historyOfTop' aligned.
top[0] is bitcoin
historyOfTop[0] is history of bitcoin

# Task 3: Make a cool chart of the history

traces = []
index = 0
for i in historyOfTop:
    x_axis = map(lambda tpls: tpls[0], i["price_usd"])
    y_axis = map(lambda tpls: tpls[1], i["price_usd"])
    matplotlib.pyplot.plot(x_axis, y_axis, label=top[index]["website_slug"])
    index = index + 1

matplotlib.pyplot.title("Prices USD")
matplotlib.pyplot.legend()
matplotlib.pyplot.show()

In the third part we draw all of the price lines in a single chart.
First we separate the X and Y.
X is the epoch timestamps.
Y is the price information
Then we plot it in matplotlib and use an 'index' to get the name of the currency from the 'top' collection
Once everything is done we show the chart.

top10coins

I hope you liked this guide.

Study hard.
Bye.

 

Getting started: bitcoin-cli

This lesson will be valid for both BTC and BCH.

Before reading the rest of it however, you might have to do some work. This lesson assumes that:

  • You know the very basics.
    • (what is a block, what is a hash and how blocks get chained)
  • You have a bitcoin node running.
    • To install a BTC node go to https://bitcoin.org/en/full-node and grab the bitcoin core client. Then start 'bitcoin-qt' to get a bitcoin node with a GUI, or 'bitcoind' for a node running as a deamon.
    • To install a BCH node go to https://www.bitcoincash.org/#nodes and grab the bitcoin abc client. Then start 'bitcoin-qt' to get a bitcoin node with a GUI, or 'bitcoind' for a node running as a deamon.

(Note 1: If you don't have 170 GB of free hard disk space on your PC, you can run the node with the '--prune' option, which tells it to delete the blocks after verifying them. This way a node can take as little space as ~500 MB)

(Note 2: Do not try to run both nodes, unless you know what you are doing. They both use the same port by default.)

 


 

Ok. Let's assume that you already have that node running.

Now let's access the bitcoin-cli. I will be using the BCH version (Bitcoin ABC node).

$ bitcoin-cli
Bitcoin ABC RPC client version v0.17.1.0-e1fa952

Usage:
bitcoin-cli [options] <command> [params] Send command to Bitcoin ABC
bitcoin-cli [options] -named <command> [name=value] ... Send command to Bitcoin ABC (with named arguments)
bitcoin-cli [options] help List commands
bitcoin-cli [options] help <command> Get help for a command

Options:
..........

From the Usage section we can see that we can pass commands, call help to get list of commands and get help for a specific command.
Make sure to always read the help for a command that you are interested in.

Now let's see what the commands are:

$ bitcoin-cli help

And here is the output:

== Blockchain ==
getbestblockhash
getblock "blockhash" ( verbose )
getblockchaininfo
getblockcount
getblockhash height
getblockheader "hash" ( verbose )
getchaintips
getchaintxstats ( nblocks blockhash )
getdifficulty
getmempoolancestors txid (verbose)
getmempooldescendants txid (verbose)
getmempoolentry txid
getmempoolinfo
getrawmempool ( verbose )
gettxout "txid" n ( include_mempool )
gettxoutproof ["txid",...] ( blockhash )
gettxoutsetinfo
preciousblock "blockhash"
pruneblockchain
verifychain ( checklevel nblocks )
verifytxoutproof "proof"

== Control ==
getinfo
getmemoryinfo
help ( "command" )
stop
uptime

== Generating ==
generate nblocks ( maxtries )
generatetoaddress nblocks address (maxtries)

== Mining ==
getblocktemplate ( TemplateRequest )
getmininginfo
getnetworkhashps ( nblocks height )
prioritisetransaction <txid> <priority delta> <fee delta>
submitblock "hexdata" ( "jsonparametersobject" )

== Network ==
addnode "node" "add|remove|onetry"
clearbanned
disconnectnode "[address]" [nodeid]
getaddednodeinfo ( "node" )
getconnectioncount
getexcessiveblock
getnettotals
getnetworkinfo
getpeerinfo
listbanned
ping
setban "subnet" "add|remove" (bantime) (absolute)
setexcessiveblock blockSize
setnetworkactive true|false

== Rawtransactions ==
createrawtransaction [{"txid":"id","vout":n},...] {"address":amount,"data":"hex",...} ( locktime )
decoderawtransaction "hexstring"
decodescript "hexstring"
fundrawtransaction "hexstring" ( options )
getrawtransaction "txid" ( verbose )
sendrawtransaction "hexstring" ( allowhighfees )
signrawtransaction "hexstring" ( [{"txid":"id","vout":n,"scriptPubKey":"hex","redeemScript":"hex"},...] ["privatekey1",...] sighashtype )

== Util ==
createmultisig nrequired ["key",...]
estimatefee nblocks
estimatepriority nblocks
estimatesmartfee nblocks
estimatesmartpriority nblocks
signmessagewithprivkey "privkey" "message"
validateaddress "address"
verifymessage "address" "signature" "message"

== Wallet ==
abandontransaction "txid"
addmultisigaddress nrequired ["key",...] ( "account" )
backupwallet "destination"
dumpprivkey "address"
dumpwallet "filename"
encryptwallet "passphrase"
getaccount "address"
getaccountaddress "account"
getaddressesbyaccount "account"
getbalance ( "account" minconf include_watchonly )
getnewaddress ( "account" )
getrawchangeaddress
getreceivedbyaccount "account" ( minconf )
getreceivedbyaddress "address" ( minconf )
gettransaction "txid" ( include_watchonly )
getunconfirmedbalance
getwalletinfo
importaddress "address" ( "label" rescan p2sh )
importmulti "requests" "options"
importprivkey "bitcoinprivkey" ( "label" ) ( rescan )
importprunedfunds
importpubkey "pubkey" ( "label" rescan )
importwallet "filename"
keypoolrefill ( newsize )
listaccounts ( minconf include_watchonly)
listaddressgroupings
listlockunspent
listreceivedbyaccount ( minconf include_empty include_watchonly)
listreceivedbyaddress ( minconf include_empty include_watchonly)
listsinceblock ( "blockhash" target_confirmations include_watchonly)
listtransactions ( "account" count skip include_watchonly)
listunspent ( minconf maxconf ["addresses",...] [include_unsafe] )
listwallets
lockunspent unlock ([{"txid":"txid","vout":n},...])
move "fromaccount" "toaccount" amount ( minconf "comment" )
removeprunedfunds "txid"
sendfrom "fromaccount" "toaddress" amount ( minconf "comment" "comment_to" )
sendmany "fromaccount" {"address":amount,...} ( minconf "comment" ["address",...] )
sendtoaddress "address" amount ( "comment" "comment_to" subtractfeefromamount )
setaccount "address" "account"
settxfee amount
signmessage "address" "message"

There are quite a few things you can do with this client.
You can read blocks and transaction, send money, handle a wallet, or even publish raw transactions, which give you the ability to write custom scripts. Explaining these scripts is outside of the scope of this article, so let's leave it for another time...

For now, let's see how the command help looks like. Let's get more info on the 'submitblock' from the mining category:

$ bitcoin-cli help submitblock

...and the output is:

submitblock "hexdata" ( "jsonparametersobject" )

Attempts to submit new block to network.
The 'jsonparametersobject' parameter is currently ignored.
See https://en.bitcoin.it/wiki/BIP_0022 for full specification.

Arguments
1. "hexdata" (string, required) the hex-encoded block data to submit
2. "parameters" (string, optional) object of optional parameters
{
"workid" : "id" (string, optional) if the server provided a workid, it MUST be included with submissions
}

Result:

Examples:
> bitcoin-cli submitblock "mydata"
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "submitblock", "params": ["mydata"] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/

As you see, the 'bitcoin-cli' program covers pretty much anything you might need to do when integrating Bitcoin in your programs. There are other tools that do the same, of course. There are libraries in almost any language. The bitcoin-cli however comes bundled with the most popular nodes. It can be considered the default client.

Take some time to read about all it can do.

Now try running these commands and analyze each row of the outputs (consider this your homework):

$ bitcoin-cli getinfo
$ bitcoin-cli getblockchaininfo
$ bitcoin-cli getblock 000000001aeae195809d120b5d66a39c83eb48792e068f8ea1fea19d84a4278a

(getblock only works if your node is not --pruned)

$ bitcoin-cli getnetworkinfo
$ bitcoin-cli getchaintips

(You might need to read the help for getchaintips very carefully. It is a bit harder to understand.)

$ bitcoin-cli getmempoolinfo

 


 

Remember that many of the settings you see in the outputs of these commands can change. Some of them can be changed by the owner of the node (like which network you use to connect to the other nodes), while others are decided by the network (like the mining difficulty).
Think about these changes. Make sure to understand what they would mean.

Study hard. Bye!