User Guides Archives - DomainTools | Start Here. Know Now.

The AXA User Guide

Don Lewis — Tue, 04 Apr 2023 20:32:14 +0000

Farsight Security’s (now a part of DomainTools) Advanced Exchange Access (AXA) is a suite of tools and library code that brings the Security Information Exchange (SIE) directly to the end-user’s network.

SIE is a scalable and adaptable real-time data streaming and information sharing platform. SIE collects and provides access to more than 200,000 observations per-second of raw data from its global sensor network. Farsight also applies unique and proprietary methods for improving usability of the data, directly sharing the refined intelligence with SIE customers and DNSDB®, one of the world’s largest passive DNS (pDNS) databases.

The diverse set of data available from SIE includes the following and is relevant and useful for practitioners in various technology roles:

Raw and processed passive DNS data
Darknet/darkspace telescope data
SPAM sources and URLs
Phishing URLs and associated targeted brands
Connection attempts from malware-infected systems (as seen by a sinkhole)
Network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices

Each unique set of data in SIE is known as a channel and the data acquired from a specific channel can be customized to meet the needs of each customer, enabling you to subscribe to and access only the channels needed to solve your problem. A channel in SIE may be the result of raw data analysis, or a subset of data from other channels.

The data available from SIE channel subscription packages includes:

Raw Passive DNS: Real-time observations of DNS cache-miss traffic sent from DNS recursive resolvers on the Internet to authoritative name servers. The DNS information includes authoritative DNS data that various zone operators make available and responses from authoritative name servers to recursive resolver queries
Value-Added Passive DNS: Real-time de-duplicated, filtered, and verified passive DNS (pDNS) data observed on the Internet
Newly Observed Domains (NOD) and Newly Observed Hostnames (NOH): Real-time actionable insights for domains and hostnames, fully qualified domain names (FQDNs), when they are first successfully resolved on the Internet
Base Channels: A collection of threat intelligence channels that provide access to honeypot data (darknet and spam) and botnet (e.g., Conficker) sinkhole data. The data also includes threat intelligence for phishing campaigns and log data for network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices
Premium Channels: A range of premium security-related feeds including malware metadata, IOCs and other telemetry. Subscribers consume the intelligence as real-time event flows rather than traditional batch transfers – which are inherently delayed

The SIE Channel Guide provides an overview of the channels available from SIE.

Advanced Exchange Access Toolkit

The Advanced Exchange Access (AXA) toolkit contains tools and a C library to bring Farsight’s real-time data and services directly from the Farsight Security Information Exchange (SIE) to the subscriber’s network edge.

AXA enables subscribers to connect to Farsight’s subscription-based SRA (SIE Remote Access) servers. These servers provide access to data and services built from Farsight’s SIE.

Contents of the Advanced Exchange Access Toolkit

Farsight freely provides several end-user tools to stream AXA data in the Advanced Exchange Access Toolkit. This distribution contains the following:

sratool: A command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data.
sratunnel: A command-line tool that streams SIE data to the local network.
libaxa: A C library providing an API for the AXA protocol including:
- connection instantiation/teardown
- message encapsulation/decapsulation
- watch parsing/loading
- control packet rate limits, sampling rates, window sizes, and many other AXA-specific functions

Many of the features and options we’ll be discussing are the same amongst all of the tools and examples will be given where appropriate.

Farsight SIE Remote Access

SIE Remote Access (SRA) is Farsight’s software solution to make SIE content available to remote users. SRA enables SIE channel traffic to be delivered through a TCP stream across the Internet. In order to reduce bandwidth, SRA provides subscribers with the ability to invoke a server-side filtering capability across a set of channels, selecting only that subset of records that match specific domain name / IP address search criteria.

The Advanced Exchange Access suite of tools and library code is the software that implements Farsight’s SRA service.

The AXA suite consists of two Unix command line tools and one C library developers can use to build custom SRA applications.

sratool

sratool is the AXA Swiss Army Knife. It is a versatile tool used to test, debug, or stream AXA connections. It connects to an SRA server, sends protocol messages and displays the responses. It can also tunnel SIE data like sratunnel does.

sratunnel

sratunnel transfers selected SIE data from the remote server to the local network. The connection to the server is created and restored, with binary exponential delays between retries.

sratunnel is the workhorse of the AXA family. It is used to transfer SIE data from the remote server to the local network. It is what Farsight uses for production deployment of SIE data to the customer. sratunnel can be thought of as a fast, efficient, and smart conduit for SIE data. Data goes in one end and sratunnel can emit the data into different output formats, including:

NMSGs to a UDP port
NMSGs to a TCP port
NMSGs to a file
pcap to a file
pcap to a network interface

libaxa

libaxa is the C programming library that exposes the AXA API to the application programmer.

The AXA Protocol

The AXA protocol is documented in the section titled AXA Protocol in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit.

AXA Limits

Some of the channels offered by the SIE network burst to an extremely high bitrate. AXA has two ways to deal with such network-hungry situations: optional filtering and loss-tolerance built into the protocol.

Filtering can take one of the following forms:

Via the rate limit option to reduce the flow of ingress data to a certain number of packets per second.
Via one or more IP-based or DNS-based “watches” to limit the flow of data to specific assets the subscriber wishes to observe.

Finally, AXA is a deliberately lossy protocol. If a subscriber requests more data than the network can carry, data overruns will occur. When this happens, loss markers are transmitted reliably within the AXA stream to inform the subscriber via the AXA accounting subsystem (see below). At this point, the subscriber’s possible mitigation strategies include:

ask for less data via rate limiting
increase network capacity and/or other host resources
treat the SRA stream as a chunky and non-representative sample of the total SIE data
pursue Direct Access to SIE

AXA Watches

In the AXA world, the way an end-user registers interest in an asset is through what’s known as a watch. These are fundamental building blocks of AXA sessions. In order to get anything done, an end-user must specify one or more watches to inform the AXA server what she is interested in seeing.

There are four different types of AXA watches, each is described below:

IP Watches: used to express interest in SIE messages containing a specified IP address or CIDR block. AXA supports both IPv4 and IPv6 address types.
DNS Watches: used to express interest in SIE messages containing a specified hostname, domain, or wildcard hostname.
Channel Watches: used to express interest in all SIE messages from a given channel. This is useful to enable “the firehose” for a given channel and ask SRA to send everything from the specified channel rather than matching IP address information or DNS names. Only valid for SRA connections.
Error watches: used to express interest in SIE messages that cannot be decoded by the server. Only valid for SRA connections and intended only for debugging.

Watches are referenced by tags. A tag is simply an arbitrary 16-bit integer label used to keep track of individual watches.

When connected to an SRA server, tags must be unique. Each watch will use a different tag.

Depending on the tool, tagging may be done explicitly by the end-user or implicitly, by the tool. Examples follow.

AXA Watch Examples: SRA

The following examples show how to watch for SIE messages on channel 204 containing the IP address 10.0.0.1, that are in the CIDR block 192.168.0/24, or that are in the *.farsightsecurity.com domain.

sratool

The end-user specifies a unique tag per watch:

$ sratool
sra> connect ...
sra> 1 watch ip=10.0.0.1
1 OK WATCH started
sra> 2 watch ip=192.168.0/24
2 OK WATCH started
sra> 3 watch dns=*.farsightsecurity.com
3 OK WATCH started
sra> channel 211 on
* OK CHANNEL ON/OFF channel ch211 on
...

sratunnel

With sratunnel, tags are generated internally so the end-user only needs to specify the -w option and the assets above:

$ sratunnel -w ip=10.0.0.1 -w ip=192.168.0/24 -w dns=*.farsightsecurity.com -s ... -o ... -c 211

SRA and AXA RESTful Interface

The Farsight Security Advanced Exchange Access (AXA) RESTful Interface adds a streaming HTTP interface on top of the AXA toolkit to enable developers of web-based applications to interface with Farsight Security’s SIE Remote Access (SRA) servers.

The SRA module facilitates the real time streaming of data from the Security Information Exchange (SIE) over HTTP using a RESTful API.

Access is controlled via an API key that is passed as the X-API-Key HTTP header.

The Farsight Security AXA RESTful Interface does not have any specific operating system requirements as it is delivered over a RESTful API. Farsight Security provides a convenience command line interface (CLI) tool that doubles as a Python extension module which is compatible with various modern operating systems. AXA REST requires HTTPS permitted outbound to axa-sie.domaintools.com. Subscribers must have purchased a service entitlement from Farsight Security and have been provisioned an API key.

SRA Internals

The SRA module facilitates the real time streaming of data from the SIE over HTTP using a RESTful API.

Channels

SRA requires the user to specify one or more SIE channels to stream. These are specified as an integer, for example `255`.

Watches

SRA requires the user to specify one or more IP watches and/or one or more DNS watches. These tell the server what to filter and send to the client. These are specified as ip=

{/CIDR} or dns=example.com, or dns=*.example.com.

Example: -w ip=192.253.0.0/16 -w dns=*.example.com

SRA Tutorial: Watch Newly Observed Domains

This tutorial assumes you’ve signed up to receive Farsight’s Newly Observed Domains (NOD) data feed to watch newly active domains and ensure your users don’t visit any newly minted – often malicious – domains. It shows you how to examine the Newly Observed Domains (NOD) feed in real time.

$ sratool
sra> conn ....
* HELLO srad v3.0.1 stage-sie-axa-1 supporting AXA protocols v1 to v2; currently using v1
* Using AXA protocol 2
* OK USER FSI-0410-10112 authorized

Invoke sratool, and use the connect command to establish a connection to the SRA server. The connection is managed via SSH, meaning all of the benefits conferred by the SSH protocol are available to sratool. Upon success, the client emits the hello string from the AXA_P_OP_HELLO message which was sent by the server and contains the server’s software version, name, and AXA protocol version.

sra> 1 watch ch=212
1 OK WATCH started

Inform the server we want to watch SIE channel 212 traffic (this is the NOD channel). The server responds with the current watch status. The watch is the most fundamental sratool command. This is how sratool “signs up” to receive data from the SRA server. As its name implies, watch sets up a watch which is a low-level primitive that tells the SRA server that the client is interested in nmsg messages or IP packets that meet one of the following criteria:

is to, from, or contains the specified address
contains the specified domain name
arrived on the specified SIE channel
are SIE messages that could not be decoded

A watch is given a tag that is an integer label used to refer to the watch. An SRA server connection or session can have zero or more watches at a time and the user can add or delete watches as needed. Note that sratool allows only a single SRA connection at a time.

sra> count 5

Using the count command, we inform sratool we only want to see 5 packets. After this number is met, sratool will stop emitting packets to the screen (though traffic may still be flowing from server to client).

sra> channel 212 on
* OK CHANNEL ON/OFF channel ch212 on

With the channel command, enable channel 212 (NOD). The current channel status is printed. Another fundamental command to sratool is channel. Issued alone on the command-line, it will emit the entire list of available SIE channels for which the user is provisioned.

1 ch212  {"time":"2023-02-01 19:10:02.575703046","vname":"SIE","mname":"newdomain","source":"a1ba02cf","message":{"domain":"meurepetidor.ws.","time_seen":"2023-02-01 19:09:36","bailiwick":"ws.","rrname":"meurepetidor.ws.","rrclass":"IN","rrtype":"MX","rdata":["1 mail.hope-mail.com."],"keys":[],"new_rr":[]}}
1 ch212  {"time":"2023-02-01 19:10:03.920603701","vname":"SIE","mname":"newdomain","source":"a1ba02cf","message":{"domain":"aqrin.me.","time_seen":"2023-02-01 19:09:04","bailiwick":"me.","rrname":"aqrin.me.","rrclass":"IN","rrtype":"NS","rdata":["carlos.ns.cloudflare.com.","davina.ns.cloudflare.com."],"keys":[],"new_rr":[]}}
1 ch212  {"time":"2023-02-01 19:10:07.132796370","vname":"SIE","mname":"newdomain","source":"a1ba02cf","message":{"domain":"speedpaw.com.","time_seen":"2023-02-01 19:08:58","bailiwick":"com.","rrname":"speedpaw.com.","rrclass":"IN","rrtype":"NS","rdata":["liz.ns.cloudflare.com.","marty.ns.cloudflare.com."],"keys":[],"new_rr":[]}}
1 ch212  {"time":"2023-02-01 19:10:09.254487282","vname":"SIE","mname":"newdomain","source":"a1ba02cf","message":{"domain":"jacquelinepongmarketing.com.","time_seen":"2023-02-01 19:09:21","bailiwick":"com.","rrname":"jacquelinepongmarketing.com.","rrclass":"IN","rrtype":"NS","rdata":["ns1.bluehost.com.","ns2.bluehost.com."],"keys":[],"new_rr":[]}}
1 ch212  {"time":"2023-02-01 19:10:09.325774830","vname":"SIE","mname":"newdomain","source":"a1ba02cf","message":{"domain":"aashraymakkar.com.","time_seen":"2023-02-01 19:09:24","bailiwick":"com.","rrname":"aashraymakkar.com.","rrclass":"IN","rrtype":"NS","rdata":["ns2.wixdns.net.","ns3.wixdns.net."],"keys":[],"new_rr":[]}}

packet count limit exceeded

sratool emits 5 NOD packets as it receives then from the server. Once the packet count limit is reached, emission stops.

sra> count
    packet printing stopped by count 97 packets ago

Issuing the count command with no arguments prints the current count status. In this case, we find 1990 NOD packets have been streamed to the client, but since we exceeded our limit, they were not emitted to the screen.

SRA Tutorial 2: Counts And Limits

Continuing in the session above, let’s tweak a few knobs and press a few buttons.

> list watches
  1 ch=ch212

The list watches command prints all of the active watches. We’ve still got one going, we’re just not emitting any packets to the screen.

> 1 delete
  1 OK STOP watch deleted

We delete the watch by referencing its tag with the delete command.

> rate
  RATE LIMITS
  unlimited per second; current value=307
  10 seconds between reports

Another handy command, rate allows us to query the rate limiter and control it. Currently, there is no rate limiting in play – packets will be emitted as quickly as they appear. For lower bandwidth channels, like NOD, this is might not be a problem. For the DNSDB channels, which are much higher bandwidth, we’ll want to limit the

rate at which those packets are sent by the server to sratool.

> rate 1
  RATE LIMITS
  1 per second; current value=2
  10 seconds between reports

Using the rate command, we set a rate limit of 1 packet per second. This will come in handy in the last part of the tutorial where we’ll examine DNSDB.

sratunnel Tutorial: Newly Observed Domains

Now that you know how to use the Newly Observed Domains (NOD) datafeed to watch newly active domains and ensure your users don’t visit any newly minted – often malicious – domains, we are going to show you how to to tunnel the data to your local network for bulk analysis.

To consume the data in this tutorial, you’ll need another Farsight’s nmsgtool utility. It is a deeply useful all-purpose tool for working with NMSGs (network messages). NMSG is the format Farsight uses to type, structure and package arbirtary data for transit. Much of Farsight’s data is packaged and delivered as NMSG. A detailed discussion of the NMSG suite will be covered in future blog series. For now it’s just important to understand that you can work with NMSGs using nmsgtool.

This tutorial will show you how to plumb the Newly Observed Domains (NOD) feed from SIE to your local network, in real time. Commands and their output are listed with discussion below.

$ sratunnel -s apikey:APIKEY@axa-sie.domaintools.com,PORT

Invoke sratunnel. The -s option instructs the tool where and how to connect. The option string should look familiar, it’s the same one used with sratool with the same intent and results. Securely connect via SSH as sra-service to axa-sie.domaintools.com.

> -c 212

The -c option sets the channel to stream. We want NOD, which is channel 212.

> -w ch=212

The -w option sets the watch : in this case, everything on channel 212.

> -o nmsg:udp:127.0.0.1,8430

Finally, we specify -o, which tells sratunnel where to put the data it streams. In the case above, we’ve set NMSGs to localhost on port 8430 and that’s where we’ll find our output. Data is now flowing. Let’s have a look…

$ nmsgtool -l 127.0.0.1/8430

We use nmsgtool to connect to the loopback address on port 8430.

> -c 20000

The -c option specifies a maximum count of payloads to capture.

> -o channel-212.txt

The -o option tells nmsgtool to write presentation output to a file.

$ head -8 channel-212.txt

Let’s examine one entry.

[98] [2014-12-16 23:31:06.438992023] [2:5 SIE newdomain] [a1ba02cf] [] []

Each NMSG datagram contains a fixed-length header containing the message size, a UTC timestamp, the message type, a 32-bit source identifier and optional SIE operator and group codes (both empty in this case).

domain: befrenshee.com.

The fresh new domain, hot off the press!

time_seen: 2014-12-16 23:28:19

How fresh? At the time of this writing, that timestamp of when the domain was observed was just over two minutes old.

rrname: befrenshee.com.
rrclass: IN (1)
rrtype: NS (2)
rdata: ns67.domaincontrol.com.
rdata: ns68.domaincontrol.com.

The DNS meta-data associated with the domain.

$ grep ^domain: channel-212.txt | awk '{print $2}' > NOD.txt

Manipulate the data however you see fit.

sratunnel Tutorial: Create a persistent connection to SIE

An example using sratunnel as a background process to stream nmsg messages from SIE Channel 255 (SIE Heartbeat Channel) to the loopback interface on port 8000.

Invoke sratunnel with the following arguments.

$ sratunnel -s 'ssh:sra-service@axa-sie.domaintools.com' -c 255
    -w ch=255 -o nmsg:udp:127.0.0.1,8000 &

Use tcpdump to confirm messages are being streamed.

$ sudo tcpdump -i lo -c 5 -nn port 8000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
11:18:41.204425 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:18:58.672776 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:19:16.312962 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:19:33.833821 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:19:51.277784 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
5 packets captured
10 packets received by filter
0 packets dropped by kernel

Bring the background process to the foreground.

$ fg

Kill the sratunnel process by pressing Control-C.

Process messages with nsmgtool

The nmsgtool program is a single tool for taking inputs from a variety of different inputs like data streams from the network, capturing data from network interfaces, reading data from files or even standard input and making NMSG payloads available to one or more outputs.

Viewing a stream of nmsg’s from sratunnel with nmsgtool

An example using sratunnel as a background process to stream nmsg messages from SIE Channel 255 (SIE Heartbeat Channel) to the loopback interface on port 8000; using nmsgtool to connect to the loopback interface and print the nmsg to the terminal in presentation format.

Invoke sratunnel with the following arguments:

$ sratunnel -s 'ssh:sra-service@axa-sie.domaintools.com' -c 255 -w ch=255 -o nmsg:udp:127.0.0.1,8000 &

Invoke nmsgtool to connect to the loopback interface on port 8000, process three payloads and print the output to the terminal using the presentation format.

$ nmsgtool -l 127.0.0.1/8000 -c 3 -o -
[23] [2017-06-28 19:53:51.844574928] [1:11 base encode] [1ba02cfd] [] []
type: TEXT
payload: 

[23] [2017-06-28 19:53:52.345241069] [1:11 base encode] [1ba02cfd] [] []
type: TEXT
payload: 

[23] [2017-06-28 19:53:52.845875978] [1:11 base encode] [1ba02cfd] [] []
type: TEXT
payload:

Bring the background process to the foreground.

$ fg

Kill the sratunnel process by pressing Control-C.

srartunnel Tutorial: Saving a stream of nmsg’s from sratunnel with nmsgtool

An example using sratunnel as a background process to stream nmsg messages from SIE Channel 255 (SIE Heartbeat Channel) to the loopback interface on port 8000; using nmsgtool to connect to the loopback interface and saving the output to a rotating set of files using the nmsgtool kicker function.

Invoke sratunnel with the following arguments:

$ sratunnel -s 'ssh:sra-service@axa-sie.domaintools.com' -c 255
    -w ch=255 -o nmsg:udp:127.0.0.1,8000 &

Invoke nmsgtool to connect to the loopback interface on port 8000, save nmsg files to disk every sixty seconds as a background process.

$ nmsgtool -l 127.0.0.1/8000 -t 60 -k '/bin/true' -w ch255 &

List the saved files using ls.

$ ls -l
total 16
-rw-r--r-- 1 demo demo 5518 Jun 28 16:03
    ch255.20170628.2002.1498698127.548592412.nmsg
-rw-r--r-- 1 demo demo 6436 Jun 28 16:04
    ch255.20170628.2003.1498698180.574404303.nmsg

Read one of the files using nmsgtool and outputting the results to the terminal in JSON:

$ nmsgtool -r ch255.20170628.2003.1498698180.574404303.nmsg -J -
{"time":"2017-06-28 20:03:02.061745882","vname":"base",
    "mname":"encode","source":"1ba02cfd",
    "message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}}
{"time":"2017-06-28 20:03:02.562045097","vname":"base",
"mname":"encode","source":"1ba02cfd",
"message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}}
{"time":"2017-06-28 20:03:03.062705039","vname":"base",
"mname":"encode","source":"1ba02cfd",
"message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}}

Bring the nmsgtool background process to the foreground.

$ fg

Kill the nmsgtool process by pressing Control-C.
Bring the sratunnel background process to the foreground.

$ fg

Kill the sratunnel process by pressing Control-C.

sratool Tutorial: Stream SIE traffic

sratool is a test/debug/instructional command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data.

This example uses sratool to emit five messages seen on SIE Channel 255 (SIE Heartbeat Channel):

$ sratool
sra> connect ssh:sra-service@axa-sie.domaintools.com: connect to an
    SRA server using the SSH transport. SSH used its keyring to prove
    the user's identity, so there was no 'password:' prompt. The
    HELLO response from the remote end displays its version number
    and the protocol level.
sra> count 5: instruct the sratool client to stop after five
    messages are output.
sra> channel 255 on: instruct the remote end to listen to SIE
    channel 255 which was OK'd by the server indicating that it is
    provisioned for this channel according to the authentication and
    authorization level.
sra> 1 watch ch=255: watch all content on channel 255 (with no
    rate limiting or filtering).

Accounting Messages

By default, sratool will return AXA accounting messages containing current counter statistics relevant to the current session. For more details on these packet counts, see the Accounting section below

Rate Limiting

Rate limiting is used to limit the rate of incoming AXA messages transmitted from the server to the client. While it works for both SRA, it is primarily useful for high bitrate SIE channels such as the DNS Errors channels.

Rate Limiting Examples

The end-user wishes to watch for all SIE messages on channel 204 (a channel with an average bandwidth of 21Mbps) but only wishes to receive (at most) 10 messages per second.

sratool

With sratool, the end-user has the option to receive rate limit loss reports. The example below specifies a rate limit value of 10 and a 5 second interval between server rate limit reports:

$ sratool
sra> connect ...
sra> rate 10 5
* OPTION RATE LIMIT
    10 per second; current value=0
    5 seconds between reports
sra> 1 watch ch=211
sra> channel 211 on
...
* MISSED
    missed 0 input packets, dropped 0 for congestion,
    dropped 57911 for rate limit, filtered 72417
    since 2016/10/11 14:25:32

sratunnel

With sratunnel, rate limiting is specified as a simple command line option:

$ sratunnel -r 10 -w ch=211 -s ... -o ... -c 211

Sampling

Sampling is a way to statistically sample a percentage of messages from the server. For example, if a value of 50 is chosen, AXA will return a uniformly distributed random sample of 50% of the messages caught by watches. Enabling sampling with any of the tools is quite straightforward.

sratool

$ sratool
sra> connect ...
sra> sample 50
* OPTION sample 50.00%
...

sratunnel

$ sratunnel -m 50 -s ...

TCP Buffer Sizing

AXA allows the end-user to get or set the TCP send buffer size used by the server. These buffers serve to accumulate outgoing data that the network stack has not yet been able to put on the wire and data that has been received from the wire but not yet read by applications. In simpler terms, if you know what you’re doing, these options allow the end-user to adjust the size of the in-kernel TCP buffers to optimize network performance. As an aside, these options do not directly adjust the TCP window size (see TCP Windows and Window Scaling for more information).

Note that internally, TCP actually allocates twice the size of the buffer requested (using the extra space for internal purposes. This is why AXA returns a buffer size that is twice the size requested. The minimum size is 1024 and the maximum is 262142. Only sratool support this option for TCP-based connections.

sratool

With sratool, getting and setting the window size is easy:

$ sratool
sra> connect ...
sra> window
* OPTION bufsize=262142
sra> window 8192
* OPTION bufsize=16384

Accounting

AXA has a series of server-side packet counters used to track traffic totals. This is the mechanism by which AXA tracks, logs, and communicates server-side packet information. This information is intended for users of SIE Remote Access (SRA).

What is Accounting?

Accounting is AXA’s way of keeping track of traffic totals. Server-side, AXA maintains a series of per-client packet counters (a full list is below). The AXA protocol message AXA_P_OP_ACCT sent from client to server is used to query this data. The feature is available from sratool via the acct command. It is also available from sratunnel via the -A command line option. Accounting messages are also logged server-side.

Accounting Glossary

The AXA accounting counters are described below.

total-congested: Packets that were unable to be forwarded to the client due to connection issues, although it could also be client load (i.e., the connection isn’t removing data from the server fast enough, or the client isn’t reading and processing the data fast enough). Note that: high server load might increase total-missed, but it would generally decrease total-congested by throttling the input to the congested pipe / process.
total-filtered: This is an SRA input counter. It measures packets/nmsgs that have been read on the input side (usually directly from the SIE network) and will be submitted to the filtering logic (i.e.: an AXA watch).
total-missed: This is an SRA loss counter. It measures packets that the SRA server failed to receive on the input side either because the daemon was too busy or because they were lost during transit. This counter tracks NMSG and pcap-based loss.
total-ratelimited: This measures packets that were dropped instead of being forwarded to the SRA client to comply with the rate limits specified for the client.
total-sent: This measures packets that have been sent to an SRA client.

Accounting with sratool

Let’s have a look at a common sratool-based example. Here sratool is used to connect to an SRA server and a “firehose” watch is set for our popular DNS Changes channel.

$ sratool
sra> ch 214 on ; 1 wa ch=214
[watch hits omitted]

This is left to run for approximately three minutes of wall clock time. Next, the output is paused and the accounting command is run.

sra> pause
* OK PAUSE output paused
sra> acct
* OK ACCOUNTING total-filtered=66360 total-missed=0 total-collected=0 total-sent=64912
    total-ratelimited=0 total-congested=0

We see that 66,360 nmsg payloads were read from the SIE channel. For the watch set, this is a good approximation of the overall packet rate. Channel 214 has an average payload rate of approximately 340 payloads per second, and over this very small three minute window we observed ~369 packets/s.
There were no missed packets. We expect there to be no packet loss on the SIE input side. If there were for such a low bandwidth channel, this would be indicative of server-side network fault or server overload.
We see that 64,912 packets were sent from the SRA server to our sratool client. This is good and what we expected.
Because there was no congestion or rate limiting, you may wonder why the number of sent packets is lower than the number of filtered packets. This is because when we stopped sending packets to the client, we didn’t stop the server from filtering them. SRA was still reading from the input channel but wasn’t forwarding any packets.

Accounting with sratunnel

Let’s have a look at another example, this time using sratunnel. Using the timeout utility, sratunnel is run for three minutes. It connects to the same SRA server and sets a firehose watch for the DNS Errors channel. Finally, the -A 180 -d option string instructs sratunnel to emit accounting statistics every 180 seconds, with the results are written to a file.

$ timeout 180 sratunnel -s tls:userm@sra-server,1021 -w "ch=220" -c 220 -A 180 -d
    -o nmsg:file:test-220.nmsg
connecting to tls:userm@sra-server,1021
ACCOUNTING total-filtered=4183437 total-missed=44 total-collected=0 total-sent=84371
    total-ratelimited=0 total-congested=4096266

This time, 4,183,437 packets were read from the SIE channel. Calculating as above, we find that this equates to approximately 23,241 payloads per second which is commensurate to channel 220’s higher average per second rate of ~24,000 payloads (it is sourced from our Passive DNS feed which has a per second payload rate of ~120,000).
44 packets were missed on the input side. This is a .001% rate of loss and well within acceptable limits.
Only 84,371 packets were sent to the client. To understand why, we look at the congestion number.
4,096,266 packets were lost due to congestion. In this case, this high number is likely due to the fact that the client is located on the other side of the country from the SIE data center where the SRA server is located. Additionally, the client’s downstream Internet connection is just plain too slow to keep up with the high rate of channel 220.

AXA Requirements

Operating System

Linux, FreeBSD or other POSIX compliant operating systems.

Hardware

The minimum hardware requirements to get started with tools in the Advanced Exchange Access Toolkit are listed below. Depending on the amount of data being processed, the resources may need to be increased accordingly.

1 CPU
1 GB Memory
1 GB Disk

Network

Tools in the Advanced Exchange Access Toolkit require permitted outbound to axa-sie.domaintools.com and over TCP using port 22.

Service Entitlement

Subscribers must have purchased a SIE service entitlement from Farsight Security and have been provisioned access using a SSH key.

Installing Advanced Exchange Access Toolkit (axa-tools)

Debian

These instructions use Debian packages created, maintained and hosted by Farsight Security.

Download the Farsight Apt signing key.

$ sudo wget -O /etc/apt/trusted.gpg.d/debian-farsightsec.gpg
    https://dl.farsightsecurity.com/debian/archive.pubkey

Add the Farsight Debian repository.

$ echo "deb http://dl.farsightsecurity.com/debian bullseye-farsightsec main"
    | sudo tee -a /etc/apt/sources.list.d/debian-farsightsec.list

Resynchronize the package index files.

$ sudo apt update

Install the Advanced Exchange Access Toolkit (axa-tools).

$ sudo apt install axa-tools

Build from Source

See the section titled Building manually in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit

Configuring Advanced Exchange Access with SSH

At the time of provisioning you would have been asked to generate a SSH key pair used for authentication. The following steps will reference this key, make sure you reference the correct directory path when configuring the key.

$ ssh-keygen -t rsa -b 4096 -C farsight_security -f ~/.ssh/farsight_security

Create or edit the SSH config file with the following:

$ vim ~/.ssh/config

Add the following:

Host axa-sie.domaintools.com
  IdentityFile ~/.ssh/farsight_security

Installing nmsgtool

If you have not already installed nmsgtool, install it via apt:

$ sudo apt install nmsgtool nmsg-msg-module-sie

Additional Information

The post The AXA User Guide appeared first on DomainTools | Start Here. Know Now..

The AXA Technical Overview

Don Lewis — Tue, 04 Apr 2023 20:31:55 +0000

The diverse set of data available from SIE includes the following and is relevant and useful for practitioners in various technology roles:

Raw and processed passive DNS data
Darknet/darkspace telescope data
SPAM sources and URLs
Phishing URLs and associated targeted brands
Connection attempts from malware-infected systems (as seen by a sinkhole)
Network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices

The data available from SIE channel subscription packages includes:

Raw Passive DNS: Real-time observations of DNS cache-miss traffic sent from DNS recursive resolvers on the Internet to authoritative name servers. The DNS information includes authoritative DNS data that various zone operators make available and responses from authoritative name servers to recursive resolver queries
Value-Added Passive DNS: Real-time de-duplicated, filtered, and verified passive DNS (pDNS) data observed on the Internet
Newly Observed Domains (NOD) and Newly Observed Hostnames (NOH): Real-time actionable insights for domains and hostnames, fully qualified domain names (FQDNs), when they are first successfully resolved on the Internet
Base Channels: A collection of threat intelligence channels that provide access to honeypot data (darknet and spam) and botnet (e.g., Conficker) sinkhole data. The data also includes threat intelligence for phishing campaigns and log data for network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices
Premium Channels: A range of premium security-related feeds including malware metadata, IOCs and other telemetry. Subscribers consume the intelligence as real-time event flows rather than traditional batch transfers – which are inherently delayed

The SIE Channel Guide provides an overview of the channels available from SIE.

Advanced Exchange Access Toolkit

AXA enables subscribers to connect to Farsight’s subscription-based SRA (SIE Remote Access) servers. These servers provide access to data and services built from Farsight’s SIE.

Contents of the Advanced Exchange Access Toolkit

Farsight freely provides several end-user tools to stream AXA data in the Advanced Exchange Access Toolkit. This distribution contains the following:

sratool: A command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data.
sratunnel: A command-line tool that streams SIE data to the local network.
libaxa: A C library providing an API for the AXA protocol including:
- connection instantiation/teardown
- message encapsulation/decapsulation
- watch parsing/loading
- control packet rate limits, sampling rates, window sizes, and many other AXA-specific functions

Many of the features and options we’ll be discussing are the same amongst all of the tools and examples will be given where appropriate.

Farsight SIE Remote Access

The Advanced Exchange Access suite of tools and library code is the software that implements Farsight’s SRA service.

The AXA suite consists of two Unix command line tools and one C library developers can use to build custom SRA applications.

sratool

sratunnel

sratunnel transfers selected SIE data from the remote server to the local network. The connection to the server is created and restored, with binary exponential delays between retries.

NMSGs to a UDP port
NMSGs to a TCP port
NMSGs to a file
pcap to a file
pcap to a network interface

libaxa

libaxa is the C programming library that exposes the AXA API to the application programmer.

The AXA Protocol

The AXA protocol is documented in the section titled AXA Protocol in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit.

AXA Limits

Filtering can take one of the following forms:

Via the rate limit option to reduce the flow of ingress data to a certain number of packets per second.
Via one or more IP-based or DNS-based “watches” to limit the flow of data to specific assets the subscriber wishes to observe.

ask for less data via rate limiting
increase network capacity and/or other host resources
treat the SRA stream as a chunky and non-representative sample of the total SIE data
pursue Direct Access to SIE

AXA Watches

There are four different types of AXA watches, each is described below:

IP Watches: used to express interest in SIE messages containing a specified IP address or CIDR block. AXA supports both IPv4 and IPv6 address types.
DNS Watches: used to express interest in SIE messages containing a specified hostname, domain, or wildcard hostname.
Channel Watches: used to express interest in all SIE messages from a given channel. This is useful to enable “the firehose” for a given channel and ask SRA to send everything from the specified channel rather than matching IP address information or DNS names. Only valid for SRA connections.
Error watches: used to express interest in SIE messages that cannot be decoded by the server. Only valid for SRA connections and intended only for debugging.

Watches are referenced by tags. A tag is simply an arbitrary 16-bit integer label used to keep track of individual watches.

When connected to an SRA server, tags must be unique. Each watch will use a different tag.

Depending on the tool, tagging may be done explicitly by the end-user or implicitly, by the tool.

SRA and AXA RESTful Interface

The SRA module facilitates the real time streaming of data from the Security Information Exchange (SIE) over HTTP using a RESTful API.

Access is controlled via an API key that is passed as the X-API-Key HTTP header.

By default, axamd will return AXA accounting messages containing current counter statistics relevant to the current session. For more details on these packet counts, see the Accounting section below

Rate Limiting

Rate limiting is used to limit the rate of incoming AXA messages transmitted from the server to the client. While it works for SRA, it is primarily useful for high bitrate SIE channels such as the DNS Errors channels.

Sampling

TCP Buffer Sizing

Accounting

What is Accounting?

Accounting Glossary

The AXA accounting counters are described below.

total-congested: Packets that were unable to be forwarded to the client due to connection issues, although it could also be client load (i.e., the connection isn’t removing data from the server fast enough, or the client isn’t reading and processing the data fast enough). Note that: high server load might increase total-missed, but it would generally decrease total-congested by throttling the input to the congested pipe / process.
total-filtered: This is an SRA input counter. It measures packets/nmsgs that have been read on the input side (usually directly from the SIE network) and will be submitted to the filtering logic (i.e.: an AXA watch).
total-missed: This is an SRA loss counter. It measures packets that the SRA server failed to receive on the input side either because the daemon was too busy or because they were lost during transit. This counter tracks NMSG and pcap-based loss.
total-ratelimited: This measures packets that were dropped instead of being forwarded to the SRA client to comply with the rate limits specified for the client.
total-sent: This measures packets that have been sent to an SRA client.

Additional Information

The post The AXA Technical Overview appeared first on DomainTools | Start Here. Know Now..

The AXA API Reference

Don Lewis — Tue, 04 Apr 2023 20:31:35 +0000

This article is intended to introduce and acquaint the reader with Farsight Security^® Inc.’s (now a part of DomainTools) AXA C API. It will introduce the libaxa C programming API and showcase a simple working example program.

This information is intended for users of SIE Remote Access (SRA). This article assumes the reader is familiar with AXA and related technologies. To brush up, on AXA, a good starting point is the AXA User Guide.

This article, geared towards intermediate-level C programmers, is by no means an exhaustive API reference. For that, the reader is directed to the accompanying Doxygen-based API manual that is available in the AXA GitHub repository. Please see the Additional Information section below. This article covers libaxa version 1.1.2.

sratesttool

sratesttool is a Unix command-line utility capable of the following:

Connecting to an SRA server
Enabling an SIE channel
Setting an AXA watch
Streaming watch hits to the console

Throughout this short series, we’ll examine how all of this done. In this first article, we’ll cover the main driver which contains the initialization code and controls the main program flow. For example:

$ sratesttool -s tls:user@sraserver,1021 -c 255 -w ch=255
Farsight Security SRA Test Tool
connecting to tls:user@sraserver,1021...
connected
parsing watch: ch=255...
parse ch=255 OK
setting watch on server...
watch set OK
parsing channel: 255...
parse 255 OK
enabling channel on server...
channel enabled OK
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:21.111788988
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:21.612488985
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:22.113173007
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:22.613782882
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:23.114434957
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:23.615050077
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:24.115665912
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:24.616344928
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:25.117007970
NMSG watch hit, channel: 255 @ 2015-07-29T19:13:25.617640972
^C10 total watch hits

Please note sratesttool is not a full implementation of the AXA protocol. It is meant as a tutorial on how to stand up a simple connection to the SRA server, issue a few commands, and stream data. sratesttool is only written to work for TLS and many libaxa code paths are not taken nor is robust error checking performed. It is intended as an entry-level program to get the reader familiar with the AXA protocol and the libaxa API.

Sratesttool.c

The code for sratesttool is contained in a single source file that can be compiled into a fully functional program. To build it, you’ll need to link against the libaxa library. You can find the source code in AXA GitHub repository. If you’re a Debian-user, there are packages ready for download as well. While we will cover the source code across several articles, the full source code is available for download from Farsight Security’s blog-code GitHub page.

Preamble

The first section contains the source code license and the standard C header file include progression:


/*
* Copyright (c) 2015 by Farsight Security, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
*    http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#include 
#include 
#include 
#include 

#include 
#include

Server-Specific Functions

The following server-specific functions wrap libaxa functions. We’ll explore all of them in detail later, for now it is sufficient to understand what they do:

srvr_connect(): Connects to the server. – srvr_cmd(): Send a command to the server and wait for a response.
srvr_wait_resp(): Wait for a response from the server.
srvr_process(): Process a message from the server.
srvr_disconnect(): Disconnects from the server.

/* connect to server
*
* returns true on success, false on failure
*/
static bool srvr_connect(void);

/* send an AXA command to the server and wait for the response
*
* returns true on success, false on failure
*/
static bool srvr_cmd(axa_tag_t, /* AXA tag or AXA_TAG_NONE */
axa_p_op_t, /* AXA opcode (original) */
const void *, /* body of message to send */
size_t, /* length of message */
axa_p_op_t); /* expected response opcode */

/* wait for a response from the server
*
* returns true on success, false on failure
*/
static bool srvr_wait_resp(axa_p_op_t, /* expected response opcode */
axa_p_op_t); /* original opcode*/

/* process a message from the server */
static void srvr_process(void);

/* disconnect from server and print a message */
static void srvr_disconnect(const char *, /* message to print */
...) /* optional varargs */
AXA_PF(1,2);

Miscellaneous Functions

Miscellaneous functions are declared next:

/* stop everything and shutdown */
static void stop(int                        /* exitcode */
                        );
/* signal handler */
static void sigterm(int                     /* signal number of caught sig */
                        );
/* print usage */
static void usage(const char *              /* program name */
                        );

Global Variables

There is a small collection of global data:

static axa_client_t client: Opaque blob, contains all client state.
static const char *server_str: The transport/username/server string.
static const char *watch_str: The watch string.
static const char *channel_str: The channel string.
static int terminated: When this value is > 0, it’s time to quit
static uint64_t hits: Number of watch hits sratesttool has observed.

static axa_client_t client;
static const char *server_str;
static const char *watch_str;
static const char *channel_str;
static int terminated;
static uint64_t hits;

Main Function Argument Processing

The initial stanza of main function is standard and uninteresting command-line argument processing:

int
main(int argc, char **argv)
{
    int opt;
    axa_emsg_t emsg;
    const char *cp;

    while ((opt = getopt(argc, argv, "c:hs:w:")) >= 0)
    {
        switch (opt)
        {
            case 'c':
                channel_str = optarg;
                break;
            case 'h':
                usage(argv[0]);
                return (EXIT_SUCCESS);
            case 's':
                server_str = optarg;
                break;
            case 'w':
                watch_str = optarg;
                break;
            default:
                usage(argv[0]);
                return (EX_USAGE);
        }
    }
    if (server_str == NULL || channel_str == NULL || watch_str == NULL)
    {
        fprintf(stderr, "-s, -c, -w are all required\n");
        return (EX_USAGE);
    }

Axa Initialization

Next, we encounter the first libaxa API function calls. The first stanza sets up the AXA logging substructure. One of the conveniences libaxa offers is an opaque “three stream” logging / syslog interface. With libaxa, you have the following logging streams:

trace: Tracing stream for server-side error messages
verror: Error stream for client-side error messages
accounting: Accounting information (packet sent/loss/transit/limit totals)

sratesttool does not use the syslog interface and as such initializes the loggers to emit messages only to stderr. For more information, see the Doxygen manual for axa_parse_log_opt().

After the logging initialization, we initialize the AXA client context. This is an opaque structure that contains all of the state required to open and maintain an SRA server connection.

Next comes the signal processing. In order to provide an orderly exit from signals that cause the termination of the sratesttool process with garbage collection and a statistics report, we catch and hand them off to our simple signal handler.

/* set the global program name (for logging) */
axa_set_me(argv[0]);
/* set the tracing stream to emit to stderr only */
AXA_ASSERT(axa_parse_log_opt(&emsg, "trace,off,stderr"));
/* set the error stream to emit to stderr only */
AXA_ASSERT(axa_parse_log_opt(&emsg, "error,off,stderr"));
/* set the accounting stream to emit to stderr only */
AXA_ASSERT(axa_parse_log_opt(&emsg, "acct,off,stderr"));

/* initialize the AXA syslog interface */
axa_syslog_init();

/* ensure all stdio FDs are open and ready for business */
axa_clean_stdio();

/* initialize the client context (including AXA IO engine) */
axa_client_init(&client);

/* catch and handle these signals for graceful exit */
signal(SIGPIPE, SIG_IGN);
signal(SIGHUP, sigterm);
signal(SIGTERM, sigterm);
signal(SIGINT, sigterm);

axa_trace_msg("Farsight Security SRA Test Tool\n");

Server Connection

At this point, our AXA client is initialized and sratesttool is ready for business. It then reaches out and tries to make a connection to the server. If something goes wrong, sratesttool will bail. More robust implementations such as sratool, sratunnel and axaclientd will attempt to reconnect using an exponential back off and retry algorithm.

A lot of core libaxa code is contained in the srvr_connect() function including:

Opening the client connection to the server.
Parsing the watch string and setting the watch on the server.
Parsing the channel string and enabling the channel on the server.
Starting the data stream.

We will explore this in more detail later.

/* try (once) to connect to the SRA server and bail on any error */
if (!srvr_connect())
{
    exit(EXIT_FAILURE);
}

Process Data

We now descend into the infinite event loop. The first check will always be to see if sratesttool needs to exit (as the result of an asynchronous event such as the user hitting ctrl-c at the console).

The call to axa_io_wait() tells libaxa to simply wait up to 10ms for input from the server. Internally, libaxa polls its list of file descriptors to look for input from the server. If the timer expires and no data is available, axa_io_wait() will return AXA_IO_BUSY which just returns control to the top of the loop. If data is waiting to be read, AXA_IO_OK will be returned and srvr_process() will be called to handle it (covered in detail later). For sratesttool, most of the time, this should be watch hits. If something goes wrong when polling, AXA_IO_ERR is returned and sratesttool will disconnect and quit.

Also of note, axa_io_wait() is called without keepalive functionality (not needed for this example) and without tunnel debugging (only used for SSH-based connections).

/* event loop where we continuously wait/send data/wait */
    for (;;)
    {
        if (terminated != 0)
        {
            stop(terminated);
        }
        /* wait up to 10ms for data from the server */
        switch (axa_io_wait(&emsg,
                    &client.io,         /* address of AXA IO engine */
                    10,                 /* wait up to this many ms */
                    false,              /* wake up to send a keepalive */
                    false))             /* pay attention to tunnel msgs */
        {
            /* AXA IO error */
            case AXA_IO_ERR:
                srvr_disconnect("%s", emsg.c);
                break;
            /* incomplete response, poll() and try again */
            case AXA_IO_BUSY:
                break;
            /* operation has finished, time to process the result */
            case AXA_IO_OK:
                srvr_process();
                break;
            default:
                AXA_FAIL("impossible axa_io_wait() result");
        }
    }
}

Connecting To The Server And Setting State

The first srvr_* function we’ll dissect is a big one, responsible for connecting to the SRA server, setting the watch and enabling the channel. First, we explain important AXA-specific data types/variables:

axa_p_watch_t watch: An AXA watch object; contains all of the state for an AXA watch specification.
axa_p_channel_t channel: An AXA channel object; contains all of the state for an AXA channel specification.
axa_emsg_t emsg: An AXA error message object; if an error occurs in a libaxa function, this will contain details on what caused it.
axa_tag_t cur_tag: A tag is an identifier used to uniquely “tag” specific events during the lifetime of an AXA session. To refer to these events, the client or server will use the tag. Some AXA messages do not require tags, in which case we will use the value AXA_TAG_NONE. Required tags must be unique during the lifetime of the corresponding client request. Some client requests such as a watch can last indefinitely and will elicit many server responses all with the same tag.

static bool
srvr_connect(void)
{
    axa_p_watch_t watch;
    size_t watch_len;
    axa_p_channel_t channel;
    axa_emsg_t emsg;
    bool res;
    axa_tag_t cur_tag;

Connect To The Server

The axa_client_open() function attempts to open a server connection and establish state for our client’s connection. The first two arguments, passed by reference, will be filled in by libaxa(emsg) if something goes wrong, client if something doesn’t). The next argument, server_str is a specially crafted text string of the following format: transport:user@server[,port].

transport: SRA supports either TLS or SSH as an encrypted transport. sratesttool only supports TLS, so the argument here should be tls.
user: The username of the user. This is provisioned by Farsight Security when your account is turned up.
server: The SRA server IP address or hostname.
port: Used only for TLS, to specify the port the SRA server listens for TLS connections.

The input socket buffer size is set to be 256 1024, a larger than normal value due to the potentially high-volume of SIE data transited by SRA. Internally, this will be requested to be set by setsockopt(..SO_RCVBUF..).

The return value from axa_client_open() is evaluated in a small switch table which checks for errors or non sequitur as per the following:

AXA_CONNECT_ERR: Something went terribly wrong, print emsg and quit.
AXA_CONNECT_TEMP: A temporary error occurred, print emsg and quit. In more robust implementations, we would probably back off and try again.
AXA_CONNECT_DONE: The connection is complete.
AXA_CONNECT_NOP: The connection is complete and an AXA_P_OP_NOP is sent to the server. This is done so the client can tell the server the version of the AXA protocol that the client is using. If need be, the server can down shifting to an old version of the protocol.
AXA_CONNECT_USER: For TCP-based socket connections, does not apply to TLS (or SSH) connections and should be an error in this case.

axa_trace_msg("connecting to %s...", server_str);
switch (axa_client_open(&emsg,      /* if func fails, will contain error */
                        &client,    /* client context */
                        server_str, /* server address string */
                        false,      /* true to enable RAD mode */
                        false,      /* true when SSH tunnel debugging */
                        256 * 1024, /* socket buffer size */
                        false))     /* true for non-blocking */
{
    /* permanent failure, connection has been closed, check emsg */
    case AXA_CONNECT_ERR:
        axa_error_msg("%s", emsg.c);
        return (false);
    /* A temporary failure, connection has been closed, check emsg.
    * In more robust implementations, we would try to reconnect to the
    * server.
    */
    case AXA_CONNECT_TEMP:
        axa_error_msg("%s", emsg.c);
        srvr_disconnect("%s", emsg.c);
        return (false);
    /* connect is complete */
    case AXA_CONNECT_DONE:
        break;
    /* non-blocking connection waiting for TCP SYN/ACK or TLS handshake */
    case AXA_CONNECT_INCOM:
        AXA_FAIL("impossible result from axa_client_open");
    /* connect is complete (incl xmit of AXA_P_OP_NOP) */
    case AXA_CONNECT_NOP:
        break;
    /* connect is complete (incl xmit of AXA_P_OP_USER) */
    case AXA_CONNECT_USER:
        srvr_disconnect("%s", emsg.c);
        return (false);
}

Pause The Output

After successfully connecting to the server but before any watches are set or channels are enabled, an AXA_P_OP_PAUSE command is sent. This “pause” command tells the server to stop sending data and to actually discard it, server side. This is a global switch the pauses data output for all channels for the current user.

This is the first time we encounter the srvr_cmd() function which is used to send commands and data to the SRA server. We’ll cover it in detail later but for now it’s important to note that it accepts an AXA tag argument:

AXA_TAG_MIN: The “minimum” tag value, this is a non-specific tag value used when we need a tag but don’t care about which one.

And two AXA opcode arguments:

AXA_P_OP_PAUSE: The opcode representing the action requested of the server. In this case to pause the data flow.
AXA_P_OP_OK: The opcode expected from the server the for the operation to be considered a success.

/* block any watch hits until we are ready */
if (!srvr_cmd(AXA_TAG_MIN, AXA_P_OP_PAUSE, NULL, 0, AXA_P_OP_OK))
{
    return (false);
}

Parse and Set the Watch

Next, the watch string is parsed with a call to axa_parse_watch(). If there is a problem parsing it, libaxa will do its best to emit a helpful error message in emsg. However, if the watch string makes no sense, the function will fail with an empty emsg.

For a verbose treatment of what SRA watches can look like, see sratool(1).

After successfully parsing the watch, it is set on the server with a call to srvr_cmd() with an AXA opcode of AXA_P_OP_WATCH and an expected server response of AXA_P_OP_OK.

cur_tag = 1;
/* parse the watch string */
axa_trace_msg("parsing watch: %s...\n", watch_str);
res = axa_parse_watch(&emsg, &watch, &watch_len, watch_str);
if (!res)
{
    if (emsg.c[0] == '0')
    {
        axa_error_msg("unrecognized "-w %s"", watch_str);
    }
    else
    {
        axa_error_msg(""-w %s": %s", watch_str, emsg.c);
    }
    return (false);
}
else
{
    axa_trace_msg("parse %s OK\n", watch_str);
}

/* set the watch on the server */
axa_trace_msg("setting watch on server...\n");
if (!srvr_cmd(cur_tag, AXA_P_OP_WATCH, &watch, watch_len, AXA_P_OP_OK))
{
    return (false);
}
axa_trace_msg("watch set OK\n");

Parse And Enable The Channel

Next, in the same manner as above, the channel string is parsed and enabled on the server.

memset(&channel, 0, sizeof(channel));
/* parse the channel string */
axa_trace_msg("parsing channel: %s...\n", channel_str);
if (!axa_parse_ch(&emsg, &channel.ch, channel_str,
            strlen(channel_str), true, true))
{
    axa_error_msg(""-c %s": %s", channel_str, emsg.c);
    return (false);
}
else
{
    axa_trace_msg("parse %s OK\n", channel_str);
}

channel.on = 1;
axa_trace_msg("enabling channel on server...\n");
if (!srvr_cmd(AXA_TAG_MIN, AXA_P_OP_CHANNEL, &channel, sizeof(channel),
    AXA_P_OP_OK))
{
    return (false);
}

Un-Pause The Output

After the channel is enabled, sratesttool is ready to start streaming watch hits. It lets the server know by sending the opcode AXA_P_OP_GO which tells the server it’s time to un-pause the output.

Now the server connection process is complete and the function returns successfully.

    if (!srvr_cmd(AXA_TAG_MIN, AXA_P_OP_GO, NULL, 0, AXA_P_OP_OK))
    {
        return (false);
    }
    axa_trace_msg("channel enabled OK\n");
    return (true);
}

Run a Command on The Server

The next function, as we’ve already seen, srvr_cmd(), is used to send opcodes and data to the SRA server. It is a wrapper to axa_client_send() with additional code to disconnect via srvr_disconnect() if a problem occurs.

We’ve already covered the tag and opcode arguments, but the srvr_cmd() function also accepts data in the form of an opaque pointer and its length. The call to axa_client_send() fires off the request to the server and, if it doesn’t fail, a call to srvr_wait_resp() is made to process the response.

static bool
srvr_cmd(axa_tag_t tag, axa_p_op_t op, const void *b, size_t b_len,
    axa_p_op_t resp_op)
{
    axa_p_hdr_t hdr;
    axa_emsg_t emsg;
    char pbuf[AXA_P_STRLEN];

    if (!axa_client_send(&emsg, &client, tag, op, &hdr, b, b_len))
    {
        srvr_disconnect("sending %s failed: %s",
                axa_p_to_str(pbuf, sizeof(pbuf), true, &hdr, b),
                emsg.c);
        return (false);
    }

    return (srvr_wait_resp(resp_op, op));
}

Wait for a Response From the Server

The main event loop where sratesttool receives input from the SRA server and decides what to do with it, called srvr_wait_resp() is covered next. The function can be broken down into three parts:

Check to see if it’s time to quit
Receive input from the server
Determine what to do with the input

Check To See If It’s Time To Quit

The first thing srvr_wait_resp() does is check to see if the global sentry value terminated is set to true. For this to be the case, one of the signals sratesttool was initialized to catch, has been caught. Most of the time, this will be SIGINT from the user pressing “ctrl-c” at the console. In this case, stop() (covered below) is called which begins the shutdown process.

static bool
srvr_wait_resp(axa_p_op_t resp_op, axa_p_op_t orig_op)
{
    bool result, done;
    axa_emsg_t emsg;

    result = false;
    done = false;
    do
    {
        if (terminated != 0)
        {
            stop(terminated);
        }

Receive Input From The Server

If it’s not time to quit, control is passed to axa_input(). This function accepts the standard emsg argument we’ve already seen, a pointer to a client.io context, and the number of milliseconds to block and wait for input which in this case is INT_MAX (this can be considered an indefinite block while waiting for data).

The result is switched and the following cases are evaluated:

AXA_IO_ERR: A fatal error occurred, dump the error and return false which will result in the upper layer terminating the program.
AXA_IO_BUSY: There was no input before the poll() timer expired or some other non-fatal condition occurred. Control continues back up to the top.
AXA_IO_OK: Valid data was received, time to figure out what to do with it.

switch (axa_input(&emsg, &client.io, INT_MAX))
        {
            case AXA_IO_ERR:
                axa_error_msg("%s", emsg.c);
                goto out;
            case AXA_IO_BUSY:
                continue;
            case AXA_IO_OK:
                break;
            default:
                AXA_FAIL("impossible axa_input() result");
        }

Determine What To Do With The Input

Next, sratesttool evaluates the received AXA protocol message header’s opcode inside of a large switch table. For posterity, all of the possible opcodes are listed and in a more robust implementation, each code path would likely be populated. In the case of sratesttool only two server responses are interesting:

AXA_P_OP_HELLO: Process the “hello” message from the server. The AXA protocol version used by the server is saved and the client attempts to adjust to a version it can understand.
AXA_P_OP_OK: Process a “result” message from the server. Here sratesttool ensures that the opcode received is the expected one.

After a processing the response, axa_recv_flush() is called to purge (free()) the AXA protocol message from the IO context.

If something went wrong, axa_client_backoff() is called which will result in the client closing the connection to the server and the shutting down of the IO context.

switch ((axa_p_op_t)client.io.recv_hdr.op)
        {
            case AXA_P_OP_NOP:
                break;
            case AXA_P_OP_HELLO:
                if (!axa_client_hello(&emsg, &client, NULL))
                {
                    axa_error_msg("%s", emsg.c);
                }
                else
                {
                    axa_trace_msg("connected OK");
                }
                break;
            case AXA_P_OP_OPT:
                done = true;
                break;
            case AXA_P_OP_OK:
                if (resp_op == client.io.recv_hdr.op &&
                        orig_op == client.io.recv_body->result.orig_op)
                {
                    result = true;
                    done = true;
                }
                break;
            case AXA_P_OP_ERROR:
                axa_error_msg("server returned error");
                done = true;
                break;
            case AXA_P_OP_MISSED:
            case AXA_P_OP_MISSED_RAD:
            case AXA_P_OP_WHIT:
            case AXA_P_OP_AHIT:
            case AXA_P_OP_WLIST:
            case AXA_P_OP_ALIST:
            case AXA_P_OP_CLIST:
                /* in this function, these are all unexpected op codes */
                break;
            case AXA_P_OP_USER:
            case AXA_P_OP_JOIN:
            case AXA_P_OP_PAUSE:
            case AXA_P_OP_GO:
            case AXA_P_OP_WATCH:
            case AXA_P_OP_WGET:
            case AXA_P_OP_ANOM:
            case AXA_P_OP_AGET:
            case AXA_P_OP_STOP:
            case AXA_P_OP_ALL_STOP:
            case AXA_P_OP_CHANNEL:
            case AXA_P_OP_CGET:
            case AXA_P_OP_ACCT:
            case AXA_P_OP_RADU:
            default:
                AXA_FAIL("impossible AXA op of %d from %s",
                    client.io.recv_hdr.op, client.io.label);
        }
        axa_recv_flush(&client.io);
    }
    while (!done);

out:
    if (!result)
    {
        /* disconnect for now if we failed to get the right response */
        axa_client_backoff(&client);
    }
    return (result);
}

Disconnect From the Server

If something egregious went wrong, sratesttool will emit an error message via axa_verror_msg() and shutdown the server connection via axa_client_backoff(). As has been mentioned before, a more robust implementation may make use of the internal backoff timers and attempt a reconnect.

Stop and Shutdown

The stop() function is called any time the interrupted sentinel evaluates to not 0. It performs an orderly shutdown, reports the number of watch hits, and then exits the program.

static void AXA_NORETURN
stop(int s)
{
    axa_client_close(&client);
    axa_io_cleanup();

    axa_trace_msg("%"PRIu64" total watch hits\n", hits);

    exit(s);
}

Read SIE Data From The Server

The function that is called from the main event loop every time SIE data is ready is called srvr_process(). First, axa_recv_buf() is called and the result is evaluated. Important to note, the axa_recv_buf() can block so if this is undesirable, use of the other server read functions covered earlier such as axa_io_wait() or axa_input(). Once an AXA_IO_OK is returned, control passes to another switch table where the response opcode is evaluated.

static void
srvr_process(void)
{
    int n;
    char buf[BUFSIZ];
    axa_emsg_t emsg;
    struct timespec ts;
    struct tm *tm_info;

    switch (axa_recv_buf(&emsg, &client.io))
    {
        case AXA_IO_OK:
            break;
        case AXA_IO_ERR:
            srvr_disconnect("%s", emsg.c);
            return;
        case AXA_IO_BUSY:
            return;             /* wait for the rest */
        case AXA_IO_KEEPALIVE:  /* NA for this example */
        case AXA_IO_TUNERR:     /* NA for this example */
            break;
    }

Process the Watch Hit

The only case sratesttool is interested in is AXA_P_OP_WHIT. When the received header opcode is a “watch hit”, sratesttool next figures out the type of watch hit being reported to the client, NMSG or IP. While sratesttool is not interested in the contents of the watch hit, it does need to extract the timestamp, which is stored in different places depending on the type. Next, a human readable string is constructed using the type of watch hit, the SIE channel it occurred on, and the timestamp of the watch hit (including nanoseconds). The hits counter is incremented and control proceeds to the end of the switch table where, as seen above, axa_recv_flush() is called to purge the AXA protocol message from the IO context.

n = 0;
    switch ((axa_p_op_t)client.io.recv_hdr.op)
    {
        case AXA_P_OP_NOP:
            break;
        case AXA_P_OP_ERROR:
            srvr_disconnect(" ");
            return;
        case AXA_P_OP_MISSED:
        case AXA_P_OP_MISSED_RAD:
            break;
        case AXA_P_OP_WHIT:
            switch (client.io.recv_body->whit.hdr.type)
            {
                case AXA_P_WHIT_NMSG:
                    ts.tv_sec = client.io.recv_body->whit.nmsg.hdr.ts.tv_sec;
                    ts.tv_nsec = client.io.recv_body->whit.nmsg.hdr.ts.tv_nsec;
                    break;
                case AXA_P_WHIT_IP:
                    ts.tv_sec =
                        AXA_P2H32(client.io.recv_body->whit.ip.hdr.tv.tv_sec);
                    ts.tv_nsec = 1000 *
                        AXA_P2H32(client.io.recv_body->whit.ip.hdr.tv.tv_usec);
                    break;
            }
            tm_info = localtime((time_t *)&ts.tv_sec);
            n = strftime(buf + n, 26, "%Y-%m-%dT%H:%M:%S", tm_info);
            snprintf(buf + n, BUFSIZ - n, ".%ld", ts.tv_nsec);
            axa_trace_msg("%s watch hit, channel: %3d @ %s\n",
                    client.io.recv_body->whit.hdr.type == AXA_P_WHIT_NMSG ?
                    "NMSG" : "IP",
                    client.io.recv_body->whit.hdr.ch,
                    buf);
            hits++;
            break;
        case AXA_P_OP_AHIT:
        case AXA_P_OP_OK:
        case AXA_P_OP_HELLO:
        case AXA_P_OP_WLIST:
        case AXA_P_OP_ALIST:
        case AXA_P_OP_OPT:
        case AXA_P_OP_CLIST:
            break;
        case AXA_P_OP_USER:
        case AXA_P_OP_JOIN:
        case AXA_P_OP_PAUSE:
        case AXA_P_OP_GO:
        case AXA_P_OP_WATCH:
        case AXA_P_OP_WGET:
        case AXA_P_OP_ANOM:
        case AXA_P_OP_AGET:
        case AXA_P_OP_STOP:
        case AXA_P_OP_ALL_STOP:
        case AXA_P_OP_CHANNEL:
        case AXA_P_OP_CGET:
        case AXA_P_OP_ACCT:
        case AXA_P_OP_RADU:
        default:
            AXA_FAIL("impossible AXA op of %d from %s",
                client.io.recv_hdr.op, client.io.label);
    }

    axa_recv_flush(&client.io);
}

Signal Handler

The signal handler simply sets the global sentinel terminated to the integral value of the signal. This allows the uppers layers to:

Know when something asynchronous has happened and it’s time to quit
Know which signal was sent to sratesttool

If a signal is sent repeatedly, it is considered urgent and sratesttool will reset the default signal handler via signal(sig, SIG_DFL) which will immediately terminate the program.

void
sigterm(int sig)
{
    terminated = sig;

    signal(sig, SIG_DFL);       /* quit early on repeated signals */
}

Usage

Finally, we conclude with a simple usage function instructing the user how to invoke sratesttool.

static void
usage(const char *name)
{
    printf("SRA Test Tool (c) 2015 Farsight Security, Inc.\n");
    printf("nUsage: %s [options]\n", name);
    printf("[-s tls:user@server,port]\n");
    printf("ttuser:      SRA username\n");
    printf("ttserver:    SRA server\n");
    printf("ttport:      TCP port (typically 1021)\n");
    printf("[-h]");
    printf("ttthis prose\n");
    printf("[-c channel]");
    printf("tenable channel (default: "255", SIE Heartbeat)\n");
    printf("[-w watch]");
    printf("tset watch (default: "ch=255", SIE Heartbeat channel)\n");
}

Conclusion

We’ve covered the internals of sratesttool, a very simple “hello world” SRA client. If you are interested in learning more, please check out the AXA distribution repository which contains sratool which are more robust implementations of SRA clients built on top of libaxa.

Additional Information

Farsight’s Advanced Exchange Access (AXA) GitHub repository
AXA downloadable packages for Debian
Farsight’s Blog Code GitHub repository
Doxygen API Generator

The post The AXA API Reference appeared first on DomainTools | Start Here. Know Now..

SIE Batch REST API Reference

Don Lewis — Tue, 17 Jan 2023 17:48:00 +0000

Conventions

Every response from the SIE Batch API will contain two status indicating keys. The first is _status which is a string that will contain either OK or NOK indicating whether the API was successful or not. The second field is _message which will contain a short description of what went wrong. If the _status is OK, then _message will be present but empty.

If _status is NOK then additional fields giving details of the exception will be present, conforming to RFC 7807 .

Exception response examples are provided for each API below where appropriate.

Authentication

Validate – Validate an API Key

Validates a given API Key, returns profile information associated with it.

POST

https://batch.sie-remote.net/siebatchd/v1/validate

Permission: public

Example cURL

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
curl -d '{"apikey":"'$APIKEY'"}' https://batch.sie-remote.net/siebatchd/v1/validate

Example JS XMLHttpRequest

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

let xhr = new XMLHttpRequest();
xhr.open("POST", "https://batch.sie-remote.net/siebatchd/v1/validate");
xhr.send(JSON.stringify({apikey: apikey}));

xhr.onload = function() {
  do_something();
};

xhr.onprogress = function(event) {
  do_something_else();
};

xhr.onerror = function() {
  do_something_error();
};

Example JS React, Axios

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

axios({
  url: "https://batch.sie-remote.net/siebatchd/v1/validate",
  method: "POST",
  data: {
    apikey: apikey,
  },
}).then(
  response => {
    if(response.status === 200) {
      do_something();
    } else {
      do_something_else();
    }
  },
  error => {
    do_something_error();
  }
);

Example Object

{
  "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

Parameter

Field	Type	Description
profile	Dictionary	A dictionary containing user profile information (provisioning). The following keys are only some of the possible keys available.
username	String	The username associated with your API Key.
siebatch	Dictionary	The SIE batch channels you have access to.
_status	String	OK
_message	String	An account status message (“” [empty] if no message).

Success-Response

HTTP/1.1 200 OK

{
  "profile": {
    "username": "siebatch-customer",
    "siebatch": {
      "ch212": {
        "description": "Newly Observed Domains"
      },
      "ch213": {
        "description": "Newly Observed Fully Qualified Domain Names"
      },
    }
  },
  "_status": "OK",
  "_message": ""
}

Error 4xx 5xx

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channels	List	List (array) of one or more channel numbers (integers). Valid channels are 1-255.

MissingAPIKey

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

Batched Channel Access

siebatch chdetails – Get Channel Details

Request the details of a channel such as possible datetime ranges, download filetype, and
the size of data available.

These details will update as data is stored and pruned on the server every minute; channel data is always in motion so the details represent a moving window of data that is available for download. This means that the size and earliest/latest times will change frequently. Because channels are continuously recording new data it’s not possible to always be up to date if you were to repeatedly request the latest “earliest” and “latest” details for a channel; it’s an archival reference only.

POST

https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails

Permission: user

Example cURL

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channels":[212, 213]}' \

-X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails

Example JS XMLHttpRequest

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

let xhr = new XMLHttpRequest();
xhr.open("POST", "https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails");
xhr.send(JSON.stringify({apikey: apikey, channels: [212, 213]}));

xhr.onload = function() {
  do_something();
};

xhr.onprogress = function(event) {
  do_something_else();
};

xhr.onerror = function() {
  do_something_error();
};

Example React, Axios

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

axios({
  url: "https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails",
  method: "POST",
  data: {
    apikey: apikey,
    channels: [212, 213]
  },
}).then(
  response => {
    if(response.status === 200) {
      do_something();
    } else {
      do_something_else();
    }
  },
  error => {
    do_something_error();
  }
);

Example Object

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channels": [212, 213]
}

Parameter

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channels	List	List (array) of one or more channel numbers (integers). Valid channels are 1-255.

Success 200

Field	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

Success Response Example

HTTP/1.1 200 OK

{
  "channels": {
    "ch212": {
      "earliest":"2020-01-28 00:40:00",
      "latest":"2020-01-28 20:06:00",
      "size":83837922,
      "mimetype": "application/x-ndjson"
    },
    "ch213": {
      "earliest":"2020-01-28 00:40:00",
      "latest": "2020-01-28 20:06:01",
      "size":10397567992,
      "mimetype": "application/x-ndjson"
    }
  },
  "_status": "OK",
  "_message":""
}

Error 4xx 5xx

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure

HTTP/1.1 403
{
    "_message": "",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters

HTTP/1.1 403
{
    "_message": "",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

siebatch chfetch – Fetch Channel Data

Download data associated with a given channel.

As mentioned in chdetails, the earliest data available is pruned frequently, so you are not guaranteed to retrieve that data if you request it.

The seconds portion of the start/end times are ignored, but for API consistency, we require seconds to be given. This means that if you ask for “12:10:30” thru “12:45:30” you will get “12:10:00” thru “12:45:00”.

If no data is available then an HTTP 404 is returned – otherwise it returns 200, sets the appropriate HTTP headers, and the download begins.

POST

https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch

Permission: User

Example cURL

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channel":213,"start_time":"2020-01-28 12:00:00","end_time":"2020-01-28 12:01:00"}' \ 
-X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch

Example JS XMLHttpRequest

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

let xhr = new XMLHttpRequest();
xhr.open("POST", "https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch");
xhr.send(JSON.stringify({apikey: apikey, channel: 213, start_time: "2020-01-28 12:00:00",end_time:"2020-01-28 12:01:00"}));

xhr.onload = function() {
  do_something();
};

xhr.onprogress = function(event) {
  do_something_else();
};

xhr.onerror = function() {
  do_something_error();
};

Example JS React, Axios

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

axios({
  url: "https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch",
  method: "POST",
  data: {
    apikey: apikey,
    channel: 213,
    start_time: "2020-01-28 12:00:00",
    end_time: "2020-01-28 12:01:00",
  },
}).then(
  response => {
    if(response.status === 200) {
      do_something();
    } else {
      do_something_else();
    }
  },
  error => {
    do_something_error();
  }
);

Example Object

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channel": 213,
   "start_time": "2020-01-28 12:00:00",
   "end_time": "2020-01-28 12:01:00"
}

Parameter

Field	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

Error 4xx 5xx

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure

HTTP/1.1 403
{
    "_message": "",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters

HTTP/1.1 403
{
    "_message": "",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

siebatch makeurl – Make URL

Generate a URL that can be used to download data associated with a channel later or independently of the SIE Batch webserver.

As mentioned in chdetails, the earliest data available is pruned frequently, so you are not guaranteed to retrieve it if it’s about to be pruned.

Using makeurl as opposed to chfetch is advantageous as generated URLs can be more easily shared and accessed later on different computers and by fellow users, independently of the user that originally requested it to be generated and without credentials being revealed. Generated URLs are the preferred method for sharing snippets of channel data.

The expire keyword allows you to specify how long the URL is valid for. The maximum is 168 hours (1 week), the minimum is 1 hour. If you specify an expiration that does not make sense then a 404 will be returned. For example, if a channel collects the last 12 hours of data and you request a URL with an expiration of 13 hours then no data would be available for the final hour that the URL is valid. As a rule of thumb, the maximum requested expiration time should be no more than chXXX[“latest”] – chXXX[“earliest”] where chXXX is the channel you’re generating URLs for.

POST

https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl

Permission: user

Example cURL

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channel":213,"start_time":"2020-01-28 12:00:00","end_time":"2020-01-28 12:01:00"}' \
-X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl

Example JS XMLHttpRequest

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

let xhr = new XMLHttpRequest();
xhr.open("POST", "https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl");
xhr.send(JSON.stringify({apikey: apikey, channel: 213, start_time: "2020-01-28 12:00:00",end_time:"2020-01-28 12:01:00"}));

xhr.onload = function() {
  do_something();
};

xhr.onprogress = function(event) {
  do_something_else();
};

xhr.onerror = function() {
  do_something_error();
};

Example JS React, Axios

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

axios({
  url: "https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl",
  method: "POST",
  data: {
    apikey: apikey,
    channel: 213,
    start_time: "2020-01-28 12:00:00",
    end_time: "2020-01-28 12:01:00"
  },
}).then(
  response => {
    if(response.status === 200) {
      do_something();
    } else {
      do_something_else();
    }
  },
  error => {
    do_something_error();
  }
);

Example Object

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channel": 213,
   "start_time": "2020-01-28 12:00:00",
   "end_time": "2020-01-28 12:01:00"
}

Parameter

Field	Type	Description
_status	String	OK
_message	String	“” (empty if no message)
url	String	The presigned shareable URL to the specified SIE data

Success 200

Field	Type	Description
_status	String	OK
_message	String	“” (empty if no message)
url	String	The presigned shareable URL to the specified SIE data

Success-Response

HTTP/1.1 200 OK

{"_status": "OK",
 "_message": "",
 "url": "https://batch-dl.sie-remote.net/range/ch213/20200128.1200.20200128.1201?
 X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=..."
}

Error 4xx 5xx

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure

HTTP/1.1 403
{
    "_message": "",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters

HTTP/1.1 403
{
    "_message": "",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

Additional Information

The post SIE Batch REST API Reference appeared first on DomainTools | Start Here. Know Now..

DomainTools' Iris and Farsight's DNSDB Integration

Don Lewis — Sat, 10 Dec 2022 00:35:44 +0000

Introduction

Since 2017, DomainTools users have been able to leverage the power of Farsight Security® Inc.’s (now a part of DomainTools) DNSDB Passive DNS service within DomainTools’s Iris Investigative Platform. DomainTools supports two DNSDB integration models:

User Supplied Key: A user can have their DNDSDB API key installed in IRIS for their use – pDNS upgrade package: A user can purchase the Domaintools pDNS upgrade package, which gives them access to DNSDB data using Domaintools API key

Once passive DNS has been activated in your Iris account an additional “pDNS” tab will appear in the bottom right hand corner of the window:

Example 1: Find all IP addresses used by the fully qualified domain name www.hmc.edu.

Click on it to see the interface shown. You’re then ready to make DNSDB passive DNS queries from that interface.

Example Queries and Associated Output

Example 2: Find All the Domain Names Using the Nameserver ns.claremont.edu.

Example 3. Some of the Diverse Record Types Seen for ***.ietf.org*- over a two day period.

Differences Between The DomainTools Integration And Typical DNSDB API Reference Client Implementations

Users who are already familiar with DNSDB will find accessing passive DNS from within Iris to be straight forward for the most part, but there are a few idiosyncrasies you’ll nonetheless want to note.

Data Sources

The Iris passive DNS integration was built with the ability to use passive DNS from more than one passive DNS provider. Users who are purchasing service through DomainTools can choose “all” sources by default, or you can select just a single specific source (such as Farsight’s DNSDB, which will always be Source “D” in the interface).

Search Interface

To choose between searching RRnames (“left-hand side” of DNS resource records) vs Rdata (“right-hand side” of DNS resource records), toggle the “Search By” arrow in the upper right area of the window:

To select left-hand side search set the arrow to point to the left (Query)
To select right-hand side search set the arrow to point to the right (Query)
Iris provides a DNSDB record type selector that lists seven enumerated record types (A, AAAA, CNAME, MX, NS, SOA, TXT), and “All”. While “All” does include additional record types (such as SPF records, SRV records, TLSA records, etc.), “All” does NOT include every potential DNS record type. For example, DNSSEC-related records are not currently included in the results displayed in the Iris passive DNS interface Right-hand side searches of individual IP addresses are supported. CIDR queries and queries of arbitrary IP address ranges (e.g., 128.223.32.10-128.223.32.45) are not currently supported
Time fencing is supported in Iris, however, only available in “loose” mode. Loose mode causes a record to be returned if the time last seen for a domain is after a user’s specified time in the request. More information on loose mode can be found at Farsight’s DNSDB Time Fencing: A Post-Attack “Time Machine” .
IRIS times are localized, while DNSDB API normally works with UTC times Iris does not currently include the ability for the user to limit results by bailiwick. For an explanation of what a bailiwick is, see What is a Bailiwick?

Results

The web-based Iris interface returns 500 results by default, and is capped at 50,000 results
Iris results are displayed in tabular form in the Iris pDNS interface; to swing from the results of one query to another query, right click (or control-click on select operating systems) on a result
Results can be sorted by clicking on a column heading in the tabular display; click again to reverse that sort
Domain results found in passive DNS RRname data can be exported to the DomainTools pivot engine. Rdata results are not currently exportable directly to the pivot engine

Pricing

For Pricing and more information about the DomainTools Iris Integration with Farsight’s DNSDB please contact: DomainTools 2101 4th Ave, Suite 1150 Seattle, WA 98121 +1-206-838-9020 sales@domaintools.com

The post DomainTools' Iris and Farsight's DNSDB Integration appeared first on DomainTools | Start Here. Know Now..

Farsight Security's Maltego DNSDB Integration

Don Lewis — Sat, 10 Dec 2022 00:35:10 +0000

DNSDB is a Passive DNS (pDNS) historical database that provides a unique, fact-based, multifaceted view of the configuration of the global Internet infrastructure. DNSDB leverages the richness of Farsight’s Security Information Exchange (SIE) data-sharing platform and is engineered and operated by leading DNS experts.

Farsight (now a part of DomainTools) collects Passive DNS data from its global sensor array. It then filters and verifies the DNS transactions before inserting them into the DNSDB, along with ICANN-sponsored zone file access download data. The end result is the highest-quality and most comprehensive Passive DNS data service of its kind – with more than 100 billion DNS records since 2010.

Farsight’s DNSDB transforms threat data into actionable, relevant threat intelligence in real time. DNSDB’s high-performance, indexed, time-series DNS intelligence data service increases the value of an organization’s existing threat intelligence and improves visibility for an organization’s security program and protect its infrastructure from current and future threats.

DNSDB makes it easy to find related domain names and IP addresses, assuming you have an initial domain name or IP address as a starting point. DNSDB can answer questions, such as:

Where did this domain name point to in the past?
What domain names are hosted on a given IP address?
What domain names use a given name server?
What fully qualified domain names exist below a delegation point?

Farsight Security have created a package of transforms allowing Maltego to retrieve related information for domains, hostnames, network addresses and ranges, and e-mail addresses. These transforms use DNSDB to find values that were observed by one of Farsight’s DNS sensors for these entities, as well as domains resolving to these entities.

The Farsight Security DNSDB transforms expand the power of Maltego by enabling correlation and contextualization with near realtime and historical DNS intelligence; also known as passive DNS data. Using the DNSDB transforms, users can expose entire networks, gain an outside-in view of their infrastructure and pivot across DNS record types including domains, IPs, NS, MX, AAAA, SOA and many more. Wildcard searches are also available to expose hostnames or Fully Qualified Domain Names (FQDNs) in the left side wildcard, associated domains in the right side wildcard, and further pivoting across IPs to expose all associated domains, FQDNs, IPs, MX, NS, and other record types.

The DNSDB Transforms for Maltego can be used in any Maltego investigation to:

Find hostnames related to network addresses
Illuminate the DNS (and other service) hosting infrastructure of an interesting domain and find other domains of interest
Find historical locations of a service identified by a hostname or domain

Maltego and Farsight’s DNSDB Transform Set

Using the DNSDB transforms users can expose entire networks, gain an outside-in view of their infrastructure and pivot across DNS record types including domains, IPs, NS, MX, AAAA, SOA and many more. Wildcard searches are also available to expose hostnames or Fully Qualified Domain Names (FQDNs) in the left side wildcard, associated domains in the right side wildcard, and further pivoting across IPs to expose all associated domains, FQDNs, IPs, MX, NS, and other record types.

Uses

The DNSDB Transforms for Maltego can be used in any Maltego investigation to:

Find hostnames related to network addresses
Illuminate the DNS (and other service) hosting infrastructure of an interesting domain, and finding other domains of interest
Finding historical locations of a service identified by a hostname or domain

Available Transforms

Transforms on domains include:

Hostnames observed within the domain, optionally restricted to A, AAAA, CNAME types
Observed nameservers (NS records) for a domain
Observed mail servers (MX records) for a domain

Transforms on hostnames include:

Domains observed using the hostname as a nameserver (NS)
Domains observed using the hostname as a mail server (MX)
TXT records observed for the hostname
SRV records observed for the hostname
Other hostnames referencing the hostname (e.g. CNAME records)

Additional transforms include:

Extracting hostnames from e-mail addresses and URLs
Finding hostnames which start with a given label “phrase”
Finding hostnames related to a network address or address range

See the document Maltego Technical Reference or the online documentation within the Maltego system for the details on all of the available transforms.

Requirements

These transforms are available to users of Maltego CE (Free community edition), Maltego Classic (User/professional edition) and Maltego XL (Enterprise edition).

Maltego users can do a limited number of transforms with restricted returned data (12-50 records) without an API key. They can also request a 30 day trial key for evaluation that offers a higher quote and relaxed results restrictions.

Maltego Client vs API Key	Limit	CE (Free)	Classic (User)	XL (Enterprise)
Free – No API Key	Queries	12 per hour	12 per hour	12 per hour
Free – No API Key	Max results	12	50	50
FSI Subscription Key	Queries	FSI Quota	FSI Quota	FSI Quota
FSI Subscription Key	Max Results	12	10K	65K

Full access to Farsight DNSDB data requires a subscription and valid API key. To request a trial or learn more about the Farsight subscription services please contact Farsight Security.

Additional Information

The post Farsight Security's Maltego DNSDB Integration appeared first on DomainTools | Start Here. Know Now..

DNSDB App for Splunk User Guide

Don Lewis — Sat, 10 Dec 2022 00:34:25 +0000

Overview

The Farsight (now a part of DomainTools) DNSDB for Splunk App℠ gives organizations of all sizes broad analysis and investigation capabilities. The primary purpose of the Farsight DNSDB for Splunk application is to add contextual information and situational awareness from DNSDB to the organization’s internal event data as managed in Splunk.

DNSDB is the most comprehensive database of passive DNS data about how IPs, domains, and Internet infrastructures interconnect and evolve. By augmenting an organization’s internal log data with real-time Internet DNS information, security teams will be better able to analyze threats and adversary infrastructure and capabilities. This will enable them to identify, detect, correlate and take action on the intelligence.

All it takes is a simple click in Splunk. With that single click, users can learn the history and infrastructure associated with a suspicious domain name or suspicious IP address, and by doing so, gain critical insights into their event data. Users can also add this capability to their existing workflow to automatically pre-populate contextual information for all IPs and domain names visited by any of their hosts.

With its global array of sensors, Farsight Security receives more than 200,000 observations per second, observations which illuminate most material changes to the global DNS. Farsight DNSDB App for Splunk users get those real-time changes the same minute they are first observed. With more than 13 billion domains and hostnames collected since 2010 – all indexed for easy searches – DNSDB enables threat intelligence teams, security analysts and incident responders to search for specific hosts or subdomains within a domain and gain immediate insight into subordinate names under base domains.

About Farsight Dnsdb For Splunk

Author: Farsight Security, Inc.
App Version: 1.1.0
Vendor Products: Farsight Security DNSDB
Has index-time operations: False
Create an index: False
Implements summarization: False

Farsight DNSDB for Splunk allows a Splunk® Enterprise user to run DNSDB queries from an included dashboard, as well as through search commands.

Scripts And Binaries

dnsdb_query.py: Common python module for performing DNSDB queries.
dnsdb_command.py: Splunk custom command for performing DNSDB queries on hostnames/IP addresses.
dnsdb_ratelimit.py: Splunk custom command which retrieves query limit information.
dnsdb_lookup.py: External lookup for querying a set of targets against DNSDB.
- Note: please see the section titled dnsdb lookup before using this.
dnsdb_validateip.py: Internal script used by dashboard to validate IP addresses.
dnsdb_flushcache.py: Internal script used by a daily scheduled search to remove outdated responses from the KV store.

Release Notes

About This Release

Version 1.1.0 of Farsight DNSDB for Splunk is compatible with: Splunk Enterprise versions: 7.0, 6.6, 6.5, 6.4, 6.3, 6.2

CIM: N/A
Platforms: Platform independent
Vendor Products: Farsight Security DNSDB
Lookup file changes: N/A

Features

Version 1.1.0 is the current release of Farsight DNSDB for Splunk. It includes the following features:

Support for various methods of querying DNSDB API:
- External lookup
- Custom command
- Dashboard
Workflow action to query any field against DNSDB
Configurable alerts when API key is nearing their daily query quota.

Third-Party Software Attributions

Version 1.1.0 of Farsight DNSDB for Splunk incorporates the following third-party software or libraries.

dnsdb-query: https://github.com/dnsdb/dnsdb-query/

Dnsdb Lookup Considerations

Each DNSDB lookup done takes time to complete. Every event that is passed to it will generate a query to DNSDB. A search for over a few thousand events may take a moment to complete.

Farsight DNSDB API access is capped at a contracted number of queries per day. Every event passed to the DNSDB lookup will count as a query towards the user’s daily quota. Please be mindful of this when using the lookup functionality so as to not accidentally exhaust your daily query limit. (Should this happen on a regular basis, the query limits can be changed to meet the needs of your threat intelligence team).

Getting Started

Pre-Installation Checklist

Before installing Farsight DNSDB App for Splunk, please ensure:

Your Search Head can access api.dnsdb.info via HTTPS (TCP port 443). If you are a Farsight DNSDB-Export customer, then your Search Head needs to access your local dnstli server.
You have administrative privileges within Splunk (our app requires particular permissions).

Software Requirements

Farsight DNSDB for Splunk can run on Windows, OS X, or Linux.

Farsight DNSDB for Splunk app has no specific additional hardware requirements.

Splunk Enterprise System Requirements

Because this add-on runs on Splunk Enterprise, all of the Splunk Enterprise system requirements apply.

Installing Farsight Dnsdb App For Splunk

Install the application within Splunk by browsing to Apps > Manage Apps > Find more apps online,

and searching for Farsight DNSDB.

Or, download the package from Splunkbase at: https://splunkbase.splunk.com/app/3050 and then upload it to your Search Head.

Follow the on-screen installation steps and then restart Splunk.

To install and configure this app on your supported platform, follow these steps:

Download app from Splunkbase located at: https://splunkbase.splunk.com/app/3050
Place app.tar.gz somewhere on your Search Head
Install using splunk command: SPLUNK INSTALL APP /PATH/TO/APP.TAR.GZ
Set the DNSDB API key provided by Farsight Security, Inc. This can be done in Splunkweb by clicking Set up on the Manage apps page, or through the command line by editing dnsdb.conf.

Here are detailed, stepwise instructions to initially set up the Farsight DNSDB for Splunk app.

From the entry screen, select the gear icon next to Apps.

Click the [Install app from file] button.

Click the [Choose File] button and select the SPL file provided by Farsight. Click [Upload]

Installation of the Farsight DNSDB App will require a restart of Splunk. If you wish to restart now, click [Restart Splunk].

Once the restart is complete, login as the Administrator user again.

Click [Set up now] to configure the Farsight DNSDB App.

Enter your Farsight API key.

To inquire about a trial API key please contact the DomainTools sales team.
If you have licensed DNSDB-Export, please change the API URI to the URI used on your export server.

Click on the Splunk> logo to return to the main screen. To access the App, click on [Farsight DNSDB for Splunk].

You are now ready to use the Farsight DNSDB for Splunk app.

Enabling Automatic Lookups In Farsight Dnsdb App For Splunk

To provide context for ALL domains and IP addresses within your Splunk instance, you can enable automatic lookups to ensure the information you may need will be immediately ready.

Please note that this will cause a high number of DNSDB queries to occur.

Instructions to enable automatic lookups:

Select Settings from the Top Menu-bar and in the Knowledge section select “Lookups”

Find “Automatic lookups”, click “Add new”

Set the following fields (see attached screenshot for detailed view):

Destination app: For this example we’re going to choose search, you
may wish to select something different.
Name: We’re going to call our lookup “ClientIP-DNSDB lookup”
Lookup table: dnsdb
Apply to: sourcetype -> access_combined. This example will make use
of combined format weblogs. Select the appropriate for your
application.
Fields: The DNSDB lookup capability has 2 fields dnsdb_host and
dnsdb_ip. One is used as input, the other will automatically be
the output.
Lookup input fields:
– dnsdb_host – When the field you wish to act on is a hostname
– dnsdb_ip – When the field you wish to act on is an IP.
- Put the appropriate lookup field in the left-hand input-box. In the
  right-hand input box enter the name of the field you wish to
  lookup in your event. In our case it is “clientip”.
Lookup output fields: If the input is a dnsdb_ip the output of
lookup will be called dnsdb_host, put that in the left-hand box.
We want the output to appear in our output as clientip_hostname
so that goes in the right-hand field.
Click [Save]

It should look something like this:

Return to the main page and open search.

Search for “.”

User Guide

Usage

Once configured, the easiest way to use this app is through the built-in DNSDB dashboard. Choose a time range, type an IP address or hostname into the target field and press enter.

Farsight DNSDB for Splunk also comes with two commands and a lookup so that you can incorporate DNSDB queries into your own searches and dashboards. Below is usage documentation for all three of them.

dnsdb Command

Runs a DNSDB query on the given target. If target is an IP address, query is RDATA. Otherwise, query is RRSET. “before” and “after” fields can be supplied optionally to limit the time range of the query.

Syntax

dnsdb target=**ip/hostname** type=**rdata/rrset**
[rrtype=**A/MX/CNAME/etc] [earliest=**time**]

[latest=**latest**]

Examples

dnsdb target=203.0.113.0/24 type="rdata"
dnsdb target="example.com" latest=1446000216

dnsdblimit command

Returns the DNSDB API query limit per day, the number of queries remaining today, as well as the time when the query limit will next reset.

Syntax

dnsdblimit

Example

dnsdblimit

Dnsdb Lookup

Runs dnsdb command on a set of targets.

Syntax

lookup dnsdb [fields]

Fields

dnsdb_host, dnsdb_ip

Example

lookup dnsdb dnsdb_ip AS srcip OUTPUT dnsdb_host

Troubleshooting

*Problem: App returns error “Authorization failed. Check API key”.

Cause: API Key is missing or incorrect.

Resolution Check that your API key is entered correctly.

*Problem: App returns error “Query limit reached”.

Cause: You have reached your query limit.

Resolution Wait until your limit resets (likely at midnight daily) until making more queries.

Support And Resources

If you need help with the DNSDB Splunk integration, please contact us.

The post DNSDB App for Splunk User Guide appeared first on DomainTools | Start Here. Know Now..

Maltego Transforms Technical Reference

Don Lewis — Sat, 10 Dec 2022 00:33:51 +0000

Farsight (Now a part of DomainTools) collects Passive DNS data from its global sensor array. It then filters and verifies the DNS transactions before inserting them into the DNSDB, along with ICANN-sponsored zone file access download data. The end result is the highest-quality and most comprehensive Passive DNS data service of its kind – with more than 100 billion DNS records since 2010.

DNSDB makes it easy to find related domain names and IP addresses, assuming you have an initial domain name or IP address as a starting point. DNSDB can answer questions, such as:

Where did this domain name point to in the past?
What domain names are hosted on a given IP address?
What domain names use a given name server?
What fully qualified domain names exist below a delegation point?

Farsight Security has created a package of transforms that allows Maltego to access the DNSDB to retrieve related information for domains, hostnames, network addresses and ranges, and e-mail addresses. DNSDB transforms expand the power of Maltego by enabling correlation and contextualization with near realtime and historical DNS intelligence.

The DNSDB Transforms for Maltego can be used in any Maltego investigation to:

Find hostnames related to network addresses
Illuminate the DNS (and other service) hosting infrastructure of an interesting domain, and find other domains of interest
Find historical locations of a service identified by a hostname or domain

Maltego and Farsight’s DNSDB Transform Set

Farsight’s DNSDB transform set allows Maltego to access the DNSDB to retrieve related information for domains, hostnames, network addresses and ranges, and e-mail addresses. DNSDB transforms expand the power of Maltego by enabling correlation and contextualization with near realtime and historical DNS intelligence, allowing Maltego to retrieve related information for domains, hostnames, network addresses and ranges, and e-mail addresses. These transforms use DNSDB to find values that were observed by one of Farsight’s DNS sensors for these entities, as well as domains resolving to these entities.

The Farsight Security DNSDB transforms expand the power of Maltego by enabling correlation and contextualization with near realtime and historical DNS intelligence; also known as passive DNS data. Using the DNSDB transforms users can expose entire networks, gain an outside-in view of their infrastructure and pivot across DNS record types including domains, IPs, NS, MX, AAAA, SOA and many more. Wildcard searches are also available to expose hostnames or Fully Qualified Domain Names (FQDNs) in the left side wildcard, associated domains in the right side wildcard, and further pivoting across IPs to expose all associated domains, FQDNs, IPs, MX, NS, and other record types.

With Maltego Transforms for Farsight, investigators can correlate and contextualize with real-time and historical DNS intelligence; also known as passive DNS data.

Using these Transforms, users can expose entire networks, gain an outside-in view of their infrastructure and pivot across DNS record types. With Wildcard searches, expose hostnames/FQDNs, associated domains and further pivoting across IPs to expose all associated domains, FQDNs, IPs, MX, NX, and other record types.

To read more click here: https://www.maltego.com/transform-hub/farsight-dnsdb/

Maltego Transforms

To DNS Name (Reverse) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API Key Here	False	True	False

Transform Meta Info

Information	Value
Display Name	To DNS Name (Reverse) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrdataCIDR	maltego.CIDR	This transform finds DNS Names linked to this Netblock by an A record
dnsdbrdataDNSName	maltego.DNSName	This transform finds records where this DNS Name is in the answer
dnsdbrdataDomain	maltego.Domain	This transform finds DNS Names where the answer is this entity’s name
dnsdbrdataIPv4Address	maltego.IPv4Address	This transform finds DNS Names linked to this IP by an A record
dnsdbrdataIPv6Address1	maltego.IPv6Address	This transform finds DNS Names linked to this IP by an AAAA record
dnsdbrdataNetblock	maltego.Netblock	This transform finds DNS Names linked to this Netblock by an A record

To DNS Names [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To DNS Names [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrdataIPv6Address	maltego.Phrase	This transform finds records matching this owner name
dnsdbrrsetDomain	maltego.Domain	This transform finds records matching this owner name
dnsdbrrsetEmail	maltego.EmailAddress	This transform finds records matching the domain of this email
dnsdbrrsetURL	maltego.URL	This transform finds records matching this hostname of this URL

To Domains (Reverse, MX) [DNSDB]

Description

This transform finds NS records where this entity’s name is the answer

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To Domains (Reverse, NS) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	dnsdbrdataMXType
Input Entities	maltego.DNSName
Output Entities	Phrase
Short Description	This transform finds NS records where this entity’s name is the answer

To DNS Records [DNSDB]

Description

This transform finds records matching this owner name

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To DNS records [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	dnsdbrrsetDNSName
Input Entities	maltego.DNSName
Output Entities	Phrase
Short Description	This transform finds records matching this owner name

To IPv6 Address [DNSDB]

Description

This transform finds AAAA records where this DNS Name matches the owner name

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To IPv6 Address [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	dnsdbrrsetDNSNameToAAAA
Input Entities	maltego.DNSName
Output Entities	Phrase
Short Description	This transform finds AAAA records where this DNS Name matches the owner name

To IP Address [DNSDB]

Description

This transform finds A records where this DNS Name matches the owner name

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To IP Address [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	dnsdbrrsetDNSNameToA
Input Entities	maltego.DNSName
Output Entities	Phrase
Short Description	This transform finds A records where this DNS Name matches the owner name

To MX Record [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To MX Record [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrrsetDNSNameToMX	maltego.DNSName	This transform finds MX records for this DNS Name
dnsdbrrsetDomainMX	maltego.Domain	This transform finds MX records for this Domain
dnsdbrrsetEmailMX	maltego.EmailAddress	This transform finds MX records for the Domain referenced in this e-mail address

To NS Record [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To NS Record [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrrsetDNSNameToNS	maltego.DNSName	This transform finds NS records for this DNS Name
dnsdbrrsetDomainNS	maltego.Domain	This transform finds NS records for this Domain

To SOA Record [DNSDB]

Description

This transform finds SOA records where this DNS Name matches the owner name

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To SOA Record [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	dnsdbrrsetDNSNameToSOA
Input Entities	maltego.DNSName
Output Entities	Phrase
Short Description	This transform finds SOA records where this DNS Name matches the owner name

To SRV Record [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To SRV Record [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	dnsdbrrsetDNSNameToSRV
Input Entities	maltego.DNSName
Output Entities	Phrase
Short Description	This transform finds SRV records where this DNS Name matches the owner name

To TXT Record [DNSDB]

Description

This transform finds TXT records where this DNS Name matches the owner name

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	To TXT Record [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	dnsdbrrsetDNSNameToTXT
Input Entities	maltego.DNSName
Output Entities	Phrase
Short Description	This transform finds TXT records where this DNS Name matches the owner name

Search child DNS Names (*., AAAA) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search child DNS Names (*., AAAA) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrrsetwclDNSNameAAAA	maltego.DNSName	This transform searches for AAAA records below the owner name in this DNS Name
dnsdbrrsetwclDomainAAAA	maltego.Domain	This transform searches for AAAA records below the owner name in this Domain

Search child DNS Names (*., CNAME) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search child DNS Names (*., CNAME) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Search child DNS Names (*.) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search child DNS Names (*.) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrrsetwclDNSName	maltego.DNSName	This transform searches for hostnames below the owner name in this DNS Name
dnsdbrrsetwclDomain	maltego.Domain	This transform searches for hostnames below the owner name in this Domain
dnsdbrrsetwclPhrase	maltego.Phrase	This transform searches for hostnames below the owner name in this Phrase

Search DNS Names (.*, AAAA) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (.*, AAAA) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrrsetwcrDNSNameAAAA	maltego.DNSName	This transform searches AAAA records under a new base domains that contain this DNS Name
dnsdbrrsetwcrDomainAAAA	maltego.Domain	This transform searches AAAA records under a new base domains that contain this Domain

Search child DNS Names (*., A) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search child DNS Names (*., A) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrrsetwclDNSNameA	maltego.DNSName	This transform searches for A records below the owner name in this DNS Name
dnsdbrrsetwclDomainA	maltego.Domain	This transform searches for A records below the owner name in this Domain

Search DNS Names (.*, AAAA) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (.*, AAAA) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrrsetwcrDNSNameAAAA	maltego.DNSName	This transform searches AAAA records under a new base domains that contain this DNS Name
dnsdbrrsetwcrDomainAAAA	maltego.Domain	This transform searches AAAA records under a new base domains that contain this Domain

Search DNS Names (.*, A) [DNSDB]

Description

This transform searches A records under a new base domains that contain this DNS Name

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (.*, A) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase
Short Description	This transform searches A records under a new base domains that contain this DNS Name

Variants

Transform Name	Input Entities
dnsdbrrsetwcrDNSNameA	maltego.DNSName
dnsdbrrsetwcrDomainA	maltego.Domain

Search DNS Names (.*, CNAME) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (.*, CNAME) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrrsetwcrDNSNameCNAME	maltego.DNSName	This transform searches CNAME records under a new base domains that contain this DNS Name
dnsdbrrsetwcrDomainCNAME	maltego.Domain	This transform searches CNAME records under a new base domains that contain this Domain

Search DNS Names (.*) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (.*) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Output Entities	Phrase

Variants

Transform Name	Input Entities	Short Description
dnsdbrrsetwcrDNSName	maltego.DNSName	This transform searches for new base domains that contain this DNS Name
dnsdbrrsetwcrDomain	maltego.Domain	This transform searches for new base domains that contain this Domain
dnsdbrrsetwcrPhrase	maltego.Phrase	This transform searches for new base domains that contain this Phrase

Search DNS Names (Reverse, File Glob) [DNSDB]

Description

This transform uses Flexible search to find rdata matching this file glob pattern

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (Reverse, File Glob) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Transform Name	flexGlobRdata
Data Source	DNSDB
Input Entities	maltego.Phrase
Output Entities	Phrase
Short Description	This transform uses Flexible search to find rdata matching this file glob pattern

Search DNS Names (Glob) [DNSDB]

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (Glob) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	flexGlob
Input Entities	maltego.Phrase
Output Entities	Phrase
Short Description	Search DNS Names (Glob) [DNSDB]

Search DNS Names (Reverse, Keyword) [DNSDB]

Description

This transform uses Flexible search to find rdata matching this keyword

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (Reverse, Keyword) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	flexKeywordRdata
Input Entities	maltego.Phrase
Output Entities	Phrase
Short Description	This transform uses Flexible search to find rdata matching this keyword

Search DNS Names (Keyword) [DNSDB]

Description

This transform uses Flexible search to find hostnames matching this keyword

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (Keyword) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	flexKeyword
Input Entities	maltego.Phrase
Output Entities	Phrase
Short Description	This transform uses Flexible search to find hostnames matching this keyword

Search DNS Names (Reverse, Regex) [DNSDB]

Description

This transform uses Flexible search to find rdata matching this regular expression

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (Reverse, Regex) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	flexRegexRdata
Input Entities	maltego.Phrase
Output Entities	Phrase
Short Description	This transform uses Flexible search to find rdata matching this regular expression

Search DNS Names (Regex) [DNSDB]

Description

This transform uses Flexible search to find hostnames matching this regular expression

Transform Settings

Display Name	Setting Type	Default Value	Optional	Popup	Authentication
API Key	string	API KEY HERE	False	True	False

Transform Meta Info

Information	Value
Display Name	Search DNS Names (Regex) [DNSDB]
Owner	Farsight Support
Author	support@farsightsecurity.com
Data Source	DNSDB
Transform Name	flexRegex
Input Entities	maltego.Phrase
Output Entities	Phrase
Short Description	This transform uses Flexible search to find hostnames matching this regular expression

The post Maltego Transforms Technical Reference appeared first on DomainTools | Start Here. Know Now..

DNSDB QRadar Integration User Guide

Don Lewis — Sat, 10 Dec 2022 00:33:04 +0000

1. Introduction

Farsight Security (now part of DomainTools) DNSDB® is the world’s largest DNS intelligence database that provides a unique, fact-based, multifaceted view of the configuration of the global Internet infrastructure. DNSDB leverages the richness of Farsight’s Security Information Exchange (SIE) data-sharing platform and is engineered and operated by leading DNS experts. Farsight collects Passive DNS data from its global sensor array. It then filters and verifies the DNS transactions before inserting them into the DNSDB, along with ICANN-sponsored zone file access download data. The end result is the highest-quality and most comprehensive DNS intelligence data service of its kind – with more than 100 billion DNS records since 2010.

This document is intended to help users how to download, install and configure the Farsight DNSDB App for IBM QRadar.

Farsight DNSDB App for IBM QRadar enables to accelerate incident response with its orchestration and automation capabilities to investigate and mitigate threats. The IBM QRadar users can view the DNS enriched data in the Offense summary page for the Offense Sources. This will help to speed up QRadar investigations and prioritise the offenses to investigate and identify the source of the suspected security breach for threat hunting.

Farsight DNSDB App allows the IBM QRadar users to perform Log activity investigation in real time by using the Right-click menu options.

2. Download

Following steps to be followed while downloading the Farsight DNSDB App.

Go to IBM Security App Exchange portal and search for Farsight DNSDB.
Select Farsight DNSDB App for IBM QRadar.
From the app description page, Click on Download button to get the zip package for installation.

Please note that you will need a active IBMid to download the Farsight DNSDB App for QRadar from IBM Security App Exchange portal.

3. Installation

Follow the procedures below to install the Farsight DNSDB App.

Log in to your IBM QRadar instance.
Go to the Admin tab and select the Extensions Management.

On the Extensions Management page click on the Add button.

On the pop-up window, Click on Browse and select the downloaded zip installation package. Check Install Immediately and click Add button.

A pop-up window will open which contains the list of all the items which are part of installation package, click on Install button to continue.
After installation, a pop-up window will open which contains the list of items which are installed.
Click on Ok button to continue.
Now the Installation is completed.

4. Configuration

Once the app is installed, you will need to configure the extension as per your requirement. To do so, follow the steps below

Go to the Admin tab, scroll down to the Apps section, and select “Farsight DNSDB” App, then click on “Configure Farsight DNSDB App” (OR) you can Go to the Farsight DNSDB Dashboard, click on “Configure App”.

On the Farsight DNSDB Configuration page, you will see the following options.

Farsight DNSDB App Settings

Farsight DNSDB URL refers to the Farsight DNSDB API Endpoint of Farsight DNSDB server

Farsight DNSDB API Key refers to the API key used to authenticate requests to Farsight DNSDB server. Enter the API key provided to you by Farsight Security. If you do not have an API key then, request an API key, please see the Farsight Security Getting Started page

Enable Proxy this option is used to define Proxy settings if you desire to Configure Proxy settings to be used by Farsight DNSDB App to connect with Farsight DNSDB Server. When you check this option you will see the section to provide Proxy Settings.
- Proxy Type: Http/HTTPS
- Proxy IP/Hostname: Give a Valid IP/Hostname
- Proxy Port: Port number to connect to

If the Proxy server requires authentication, specify a username and password to connect the proxy server that requires authentication.

Automatic Offense Enrichment Settings

Look Back (days) refers to Limit records to those seen within this number of days before the offense. Default is set to 90 days.

Look Ahead(days) refers to Limit records to those seen within this number of days after the offense. Default is set to 01 days.

Limit refers to the Limit for the number of results returned per query. Default is set to 10.

Pivot Limit refers to the Limit the number of intermediate queries for co-located hosts and Ips. Default is set to 10.

Enable RData(IP Lookup) this option is used to display RData results for the IP address in the Offense Summary Page. Default is set to True.

Enable Co-located(IP Lookup) this option is used to display Co-located IP’s results for the IP address in the Offense Summary Page. Default is set to True.

Enable RData(Domain Lookup) this option is used to display RData results for the Domain in the Offense Summary Page. Default is set to True.\

Enable RRSet(Domain Lookup) this option is used to display RRSet results for the Domain in the Offense Summary Page. Default is set to True.

Enable Co-located(Domain Lookup) this option is used to display Co-located Domains results for the Domain in the Offense Summary Page. Default is set to True.

5. Automated Threat Hunting

Farsight DNSDB App enables the IBM QRadar users to view the Farsight DNSDB Passive DNS Enrichment data in the Offense summary page for the Offense Sources of type Domain/IP Address.

For IBM QRadar Offenses with IP address as Offense Source, you will see “Farsight DNSDB RData” and “Farsight DNSDB Co-located IP’s” results in a tabular format.

For IBM QRadar Offenses with Domain as Offense Source, you will see “Farsight DNSDB RData”, “Farsight DNSDB RRSET” and “Farsight DNSDB Co-located Domains” results in a tabular format.

Note: The results returned will be based on your configuration settings.

To view the Farsight DNSDB Threat Lookup results on the Offense Summary Page, Go to “Offenses” tab from the menu options.

In order to investigate, Double-click any Offense, where Offense Source is either Domain or IP Address, You will see the Farsight DNSDB Enrichment data in the tabular format.

DNSDB RDATA Results: This lookup queries DNSDB’s RData index, which supports “inverse” lookups based on RData record values. In contrast to the RRSet lookup method, RData lookups return only individual resource records and not full resource record sets and lack bailiwick metadata. An RRSet lookup on the owner name reported via an RData lookup must be performed to retrieve the full RRSet and bailiwick.

DNSDB RRSET Results:

This lookup queries DNSDB’s RRSet index, which supports “forward” lookups based on the owner name of an RRSet.

DNSDB Co-located Domain’s: This lookup will identify all the Domains that are co-located (based on Address) based on the Offense Source value. This would be set of Domains that also shared the same IP address as the originating domain name.

DNSDB Co-located IP’s: This lookup will identify all the IPs that are co-located (based on Domain) based on the Offense Source value. This would be set of IPs that also shared the same Domain as the originating IP address.

6. Manual Threat Hunting

Log Activity Investigation

Farsight DNSDB App enables the IBM QRadar users to perform Log activity investigation with Right-click menu options. When you right-click on IP Address/Domain fields in the log viewer or event viewer you will see “Lookup in Farsight DNSDB Scout” option, when you click it, this will redirect you to the “Farsight DNSDB Scout” Page, where you can make queries for the IP Address/Domain.

For any other fields apart from IP Address/Domain, select the Log Record in the log viewer or event viewer and click on “Lookup in Farsight DNSDB Scout” toolbar button, which opens a new window where you will see all the log record fields, you can click the desired field to lookup, which will redirect to the “Farsight DNSDB Scout” Page.

Threat Lookup via Dashboard

Farsight DNSDB App enables the IBM QRadar users to do Threat Lookup for user provided text irrespective of Log activity (or) offense Sources.

The next step is, go to “Farsight DNSDB Dashboard”, in the dashboard page provide any user provided text you want to search in Farsight DNSDB Scout.

The post DNSDB QRadar Integration User Guide appeared first on DomainTools | Start Here. Know Now..

Farsight Resilient Integration User Guide

Don Lewis — Sat, 10 Dec 2022 00:32:01 +0000

About the Farsight Integration for IBM’s Resilient System

Farsight DNSDB integration for IBM Resilient SOAR platform is a collection of functions that connect to DNSDB API. These functions are also accompanied with Resilient workflows that demonstrate the use of each function. These workflows can then be triggered by Resilient rules to enrich the Indicators of Compromise (IoC) with Passive DNS Data provided by DNSDB API. The extension package is bundled with all of these components (functions, workflows, rules).

DNSDB is an artifact enrichment solution. Queries are possible for:

IP Addresses
DNS Names

App Host Installation

All the components for running DNSDB function in a container already exist when using the App Host app.

To install,

Navigate to Administrative Settings and then the Apps tab.
Click the Install button and select the downloaded file: app-fn_dnsdb-x.x.x.zip.
Go to the Configuration tab and edit the app.config file and provide the DNSDB API key and Server URL.

[fn_dnsdb]
apikey =
server =

Integration Server Installation

Resilient platform >= v37.2.49
An Integration Server running resilient_circuits>=30.0.0
- To set up an Integration Server see: https://ibm.biz/res-int-server-guide
- If using API Keys, minimum required permissions are:
  - Org Data: Read, Edit
  - Function: Read

Installation

Download the fn_dnsdb.zip.
Copy the .zip to your Integration Server and SSH into it.
Unzip the package:

$ unzip fn_dnsdb-x.x.x.zip

Change Directory into the unzipped directory:

$ cd fn_dnsdb-x.x.x

Install the package:

$ pip install fn_dnsdb-x.x.x.tar.gz

Import the configurations into your app.config file:

$ resilient-circuits config -u -l fn-dnsdb

Import the fn_dnsdb customizations into the Resilient platform:

$ resilient-circuits customize -y -l fn-dnsdb

Open the config file, scroll to the bottom and edit your fn_dnsdb configurations:

$ nano ~/.resilient/app.config

Config	Required	Example	Description
apikey	Yes	d41d8cd98f00b204e9800998ecf8427e	DNSDB API key
server	Yes	https://api.dnsdb.info	DNSDB API Server

Save and Close the app.config file.
[Optional]: Run selftest to test the Integration you configured:

$ resilient-circuits selftest -l fn-dnsdb

Run resilient-circuits or restart the Service on Windows/Linux:

$ resilient-circuits run

Uninstall the Integration

SSH into your Integration Server.
Uninstall the package:

$ pip uninstall fn-dnsdb

Open the config file, scroll to the [fn_dnsdb] section and remove the section or prefix # to comment out the section.
Save and Close the app.config file.

Troubleshooting

There are several ways to verify the successful operation of a function.

Resilient Action Status

When viewing an incident, use the Actions menu to view Action Status.
By default, pending and errors are displayed.
Modify the filter for actions to also show Completed actions.
Clicking on an action displays additional information on the progress made or what error occurred.

Resilient Scripting Log

A separate log file is available to review scripting errors.
This is useful when issues occur in the pre-processing or post-processing scripts.
The default location for this log file is: /var/log/resilient-scripting/resilient-scripting.log.

Resilient Logs

By default, Resilient logs are retained at /usr/share/co3/logs.
The client.log may contain additional information regarding the execution of functions.

Resilient-Circuits

The log is controlled in the .resilient/app.config file under the section [resilient] and the property logdir.
The default file name is app.log.
Each function will create progress information.
Failures will show up as errors and may contain python trace statements.

Support

Name	Version	Author	Support URL
fn_dnsdb	1.0.0	Farsight Security, Inc.	https://farsightsecurity.com

Use Cases

DNSDB Co-Located Hosts

This use case describes the desire to easily identify Hosts that are co-located (based on Address) based on the input of a Host and a given point in time. The response would be a set of domains that also shared the same IP.

Example Pre-Process Script:

if incident.start_date:
  time_first_before = str(incident.start_date/1000 + 86400)
  time_last_after = str(incident.start_date/1000 - 15552000)
else:
  time_first_before = str(incident.create_date/1000 + 86400)
  time_last_after = str(incident.create_date/1000 - 15552000)


inputs.pivot = """
[
\{\{"function": "rrset", "owner_name": "{0}", "rrtype": "{1}", "limit": {2}, \
    "time_first_before": "{3}", "time_last_after": "{4}"}},
\{\{"function": "rdata", "type": "ip", "pivot": "rdata"}}
]
""".format(artifact.value, "A", 10, time_first_before, time_last_after)

Example Post-Process Script:

ip_records = workflow.properties.a_records["dnsdb_records"] +
    workflow.properties.aaaa_records["dnsdb_records"]

rrname_list = []

for item in ip_records:
  rrname_list.append(item["rrname"])


c_rdata_list = []

for item in workflow.properties.cname_records["dnsdb_records"]:
  for i in item["rdata"]:
    c_rdata_list.append(i)

aggregate_results = rrname_list + c_rdata_list

hosts_list_set = set(aggregate_results)
msg = ""
for item in hosts_list_set:
  msg += "{}".format(item)
msg += ""

incident.addNote(helper.createRichText("{1} Co-Located Hosts Found for \
    {0}{2}".format(artifact.value, str(len(hosts_list_set)), msg)))

DNDB Historical Address

This use case describes the desire to identify all Addresses used as DNS A and AAAA records for a given Host based on a time window from a starting and stopping point in time.

Example Pre-Process Script:

inputs.owner_name = artifact.value
inputs.limit = 100
inputs.rrtype = 'A'

#calculate time_first_before, time_last_after based on incident occur date.
if incident.start_date:
  inputs.time_first_before = str(incident.start_date/1000 + 86400)
  inputs.time_last_after = str(incident.start_date/1000 - 15552000)
else:
  inputs.time_first_before = str(incident.create_date/1000 + 86400)
  inputs.time_last_after = str(incident.create_date/1000 - 15552000)

Example Post-Process Script:

rdata_records = workflow.properties.a_records["dnsdb_records"] +
    workflow.properties.aaaa_records["dnsdb_records"]
rdata_list = []

for item in rdata_records:
  for i in item["rdata"]:
    rdata_list.append(i)

rdata_list_set = set(rdata_list)
msg = ""
for item in rdata_list_set:
  msg += "{}".format(item)
msg += ""

incident.addNote(helper.createRichText("{1} Historical Address Found for \
    {0}{2}".format(artifact.value, str(len(rdata_list_set)), msg)))

DNSDB Historical Hosts

This use case describes the desire to identify all Hosts that resolved to a given Address based on a time window from a starting and stopping point in time.

Example Pre-Process Script:

inputs.value=artifact.value
inputs.type = 'ip'
inputs.rrtype = 'ANY'
inputs.limit = 100

#calculate time_first_before, time_last_after based on incident occur date.
if incident.start_date:
  inputs.time_first_before = str(incident.start_date/1000 + 86400)
  inputs.time_last_after = str(incident.start_date/1000 - 15552000)
else:
  inputs.time_first_before = str(incident.create_date/1000 + 86400)
  inputs.time_last_after = str(incident.create_date/1000 - 15552000)

Example Post-Process Script:

dnsdb_records = results["dnsdb_records"]

host_list = []
for item in dnsdb_records:
  host_list.append(item["rrname"])

host_names_msg = ""
for item in set(host_list):
  host_names_msg += "{}".format(item)
host_names_msg += ""

incident.addNote(helper.createRichText("{1} Historical Hosts Found for: \
    {0}{2}".format(artifact.value, str(len(set(host_list))), host_names_msg)))

Function – DNSDB Flex

DNSDB Flex function allows you to search DNSDB by regular expressions and globs (aka wildcarding).

DNSDB Flex Reference Guide

Inputs:

Name	Type	Required	Description
key	select	Yes	The search key, can be rrnames(supports “forward” searches based on the owner name of an RRSet) or rdata(supports “inverse” searches based on RData record values).
limit	number	No	Limit for the number of results returned via these lookup methods.
method	select	Yes	The search method, it can be regex(regular expression search) or glob(full wildcarding).
offset	number	No	How many rows to offset (e.g. skip) in the results.
rrtype	text	No	The resource record type of the RRSet, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal rrtype number.
time_first_after	text	No	Provide results after the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_after=-31536000” will only provide results that were first observed within the last year.
time_first_before	text	No	Provide results before the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_before=1420070400” will only provide matching DNS records that were first observed before (or older than) January 1, 2015.
time_last_after	text	No	Provide results after the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_after=-2678400” will only provide results that were last observed after 31 days ago.
time_last_before	text	No	Provide results before the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_before=1356998400” will only provide results for DNS records that were last observed before 2013.
value	text	Yes	The value to search DNSDB records.

Outputs:

results = {
        "dnsdb_records": [
            {
                "from_zone_file": false,
                "rrname": "farsightsecurity.yahoo.com.au",
                "rrtype": "CNAME"
            },
            {
                "from_zone_file": false,
                "rrname": "farsightsecurity-2432183.starbucks.com.cn",
                "rrtype": "A"
            },
            {
                "from_zone_file": false,
                "rrname": "farsightsecurity.com.cn",
                "rrtype": "NS"
            },
            {
                "from_zone_file": false,
                "rrname": "farsightsecurity.com.cn",
                "rrtype": "SOA"
            },
            {
                "from_zone_file": false,
                "rrname": "farsightsecurity.com.cn",
                "rrtype": "CNAME"
            },
            {
                "from_zone_file": false,
                "rrname": "www.farsightsecurity.com.cn",
                "rrtype": "CNAME"
            },
            {
                "from_zone_file": false,
                "rrname": "farsightsecurity.damai.cn",
                "rrtype": "CNAME"
            }
        ]
}

Workflows

Example Pre-Process Script:

inputs.value = artifact.value

Example Post-Process Script:

dnsdb_records = results['dnsdb_records']

incident.addNote(helper.createRichText("{0} DNSDB Flex Records \
    Found".format(str(len(dnsdb_records)))))

Function – DNSDB RData

This function queries DNSDB’s RData index, which supports “inverse” lookups based on RData record values. In contrast to the RRSet lookup method, RData lookups return only individual resource records and not full resource record sets, and lack bailiwick metadata. An RRSet lookup on the owner name reported via an RData lookup must be performed to retrieve the full RRSet and bailiwick.

See Farsight DNSDB API Version 2 Documentation section on “rdata lookups”.

Inputs:

Name	Type	Required	Tooltip
aggr	boolean	No	Aggregated results group identical RRSets across all time periods and is the classic behavior from querying the DNSDB.
limit	number	No	Limit for the number of results returned via these lookup methods.
offset	number	No	How many rows to offset (e.g. skip) in the results.
rrtype	text	No	The resource record type of the RRSet, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal rrtype number.
time_first_after	text	No	Provide results after the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_after=-31536000” will only provide results that were first observed within the last year.
time_first_before	text	No	Provide results before the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_before=1420070400” will only provide matching DNS records that were first observed before (or older than) January 1, 2015.
time_last_after	text	No	Provide results after the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_after=-2678400” will only provide results that were last observed after 31 days ago.
time_last_before	text	No	Provide results before the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_before=1356998400” will only provide results for DNS records that were last observed before 2013.
type	select	Yes	The type specifies how value is interpreted. type may be name, ip or raw
value	text	Yes	The value to search DNSDB records.

Outputs:

results = {
        "dnsdb_records": [
            {
                "count": 606,
                "from_zone_file": false,
                "rdata": [
                    "www.farsightsecurity.com."
                ],
                "rrname": "scout.dnsdb.info",
                "rrtype": "CNAME",
                "time_first": "2020-03-27T18:37:24Z",
                "time_last": "2020-10-21T18:11:47Z"
            },
            {
                "count": 121,
                "from_zone_file": false,
                "rdata": [
                    "www.farsightsecurity.com."
                ],
                "rrname": "scout-beta.dnsdb.info",
                "rrtype": "CNAME",
                "time_first": "2020-08-20T22:52:29Z",
                "time_last": "2020-10-20T17:54:58Z"
            },
            {
                "count": 546,
                "from_zone_file": false,
                "rdata": [
                    "www.farsightsecurity.com."
                ],
                "rrname": "81.64-26.140.160.66.in-addr.arpa",
                "rrtype": "PTR",
                "time_first": "2013-12-10T01:20:08Z",
                "time_last": "2020-10-10T15:47:19Z"
            }
        ]
}

Workflows

Example Pre-Process Script:

inputs.value = artifact.value

Example Post-Process Script:

dnsdb_records = results['dnsdb_records']

incident.addNote(helper.createRichText("{0} DNSDB RData Records \
    Found".format(str(len(dnsdb_records)))))

Function – DNSDB RRSet

This function queries DNSDB’s RRSet index, which supports “forward” lookups based on the owner name of an RRSet.

See Farsight DNSDB API Version 2 Documentation section on “rrset lookups”.

Inputs:

Name	Type	Required	Tooltip
aggr	boolean	No	Aggregated results group identical RRSets across all time periods and is the classic behavior from querying the DNSDB.
bailiwick	text	No	The “bailiwick” of an RRSet in DNSDB observed via passive DNS replication is the closest enclosing zone delegated to a nameserver which served the RRset. The “bailiwick” of an RRSet in DNSDB observed in a zone file is simply the name of the zone containing the RRSet.
limit	number	No	Limit for the number of results returned via these lookup methods.
offset	number	No	How many rows to offset (e.g. skip) in the results.
owner_name	text	Yes	DNS name specified in DNS presentation format.
rrtype	text	No	The resource record type of the RRSet, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal rrtype number.
time_first_after	text	No	Provide results after the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_after=-31536000” will only provide results that were first observed within the last year.
time_first_before	text	No	Provide results before the defined timestamp for when the DNS record was first observed. For example, the URL parameter “time_first_before=1420070400” will only provide matching DNS records that were first observed before (or older than) January 1, 2015.
time_last_after	text	No	Provide results after the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_after=-2678400” will only provide results that were last observed after 31 days ago.
time_last_before	text	No	Provide results before the defined timestamp for when the DNS record was last observed. For example, the URL parameter “time_last_before=1356998400” will only provide results for DNS records that were last observed before 2013.

Outputs:

results = {
        "dnsdb_records": [
            {
                "bailiwick": "com",
                "count": 19,
                "from_zone_file": true,
                "rdata": [
                    "ns.lah1.vix.com.",
                    "ns1.isc-sns.net.",
                    "ns2.isc-sns.com.",
                    "ns3.isc-sns.info."
                ],
                "rrname": "farsightsecurity.com",
                "rrtype": "NS",
                "time_first": "2013-06-30T16:21:41Z",
                "time_last": "2013-07-18T16:22:47Z"
            },
            {
                "bailiwick": "com",
                "count": 157,
                "from_zone_file": true,
                "rdata": [
                    "ns.sjc1.vix.com.",
                    "ns.sql1.vix.com."
                ],
                "rrname": "farsightsecurity.com",
                "rrtype": "NS",
                "time_first": "2013-01-24T17:18:05Z",
                "time_last": "2013-06-29T16:19:01Z"
            },
            {
                "bailiwick": "com",
                "count": 1890,
                "from_zone_file": true,
                "rdata": [
                    "ns5.dnsmadeeasy.com.",
                    "ns6.dnsmadeeasy.com.",
                    "ns7.dnsmadeeasy.com."
                ],
                "rrname": "farsightsecurity.com",
                "rrtype": "NS",
                "time_first": "2013-07-19T16:22:00Z",
                "time_last": "2020-07-24T16:02:05Z"
            },
            {
                "bailiwick": "farsightsecurity.com",
                "count": 6350,
                "from_zone_file": false,
                "rdata": [
                    "66.160.140.81"
                ],
                "rrname": "farsightsecurity.com",
                "rrtype": "A",
                "time_first": "2013-09-25T15:37:03Z",
                "time_last": "2015-04-01T06:17:25Z"
            },
            {
                "bailiwick": "farsightsecurity.com",
                "count": 36770,
                "from_zone_file": false,
                "rdata": [
                    "104.244.13.104"
                ],
                "rrname": "farsightsecurity.com",
                "rrtype": "A",
                "time_first": "2015-04-01T14:17:52Z",
                "time_last": "2018-09-27T00:29:43Z"
            }
        ]
}

Workflows

Example Pre-Process Script:

inputs.owner_name = artifact.value

Example Post-Process Script:

dnsdb_records = results['dnsdb_records']

incident.addNote(helper.createRichText("{0} DNSDB RRSet Records Found".format(str(len(dnsdb_records)))))

Function – DNSDB Rate Limit

This function queries the “rate limit” endpoint to obtain information about the api key’s service limits and quotas.

See [Farsight DNSDB API Version 2 Documentation]({{ site.baseurl }}/dnsdb/dnsdb-apiv2/#service-limits-and-quotas) section on “service limits and quotas”.

Function – DNSDB Pivot

This function allows you to execute a series of queries against DNSDB. Each subsequent search “pivots” on the keys returned by the previous. You may pivot using the rrname, rdata, or raw_rdata field. This function takes a single argument, “pivot”, which is a json-encoded array of dictionaries. Each dictionary contains parameters for a DNSDB search, including a “function” field which may be any of “rdata”, “rrset”, or “flex”. Arguments for each search are the same as their corresponding Resilient function.

Required arguments:

rrset: owner_name
rdata: type, value
flex: key, method, value

Optional arguments:

limit, offset, time_first_before, time_first_after, time_last_before, time_last_after

The second through last dictionaries must each contain a “pivot” key which may be “rrname”, “rdata”, or “raw_rdata”, and indicates which field from the preceding query should be used as the key for the next.

Example:

inputs.pivot = """
[
\{\{"function": "rrset", "owner_name": "{0}", "rrtype": "{1}", "limit": {2},
    "time_first_before": "{3}", "time_last_after": "{4}"}},
\{\{"function": "rdata", "type": "ip", "pivot": "rdata"}}
]
""".format(artifact.value, "A", 10, time_first_before, time_last_after)

Additional Information

Rules

Rule Name	Object	Workflow Triggered
DNSDB RRSet Lookup	artifact	rrset_workflow
DNSDB RData Lookup	artifact	rdata_workflow
DNSDB Flex Lookup	artifact	flex_workflow
DNSDB Co-Located Host Lookup	artifact	dnsdb_colocated_host_workflow
DNSDB Historical Address Lookup	artifact	dnsdb_historical_address_workflow
DNSDB Historical Hosts Lookup	artifact	dnsdb_historical_host_workflow

The post Farsight Resilient Integration User Guide appeared first on DomainTools | Start Here. Know Now..