├── .gitignore ├── .editorconfig ├── docs ├── images │ ├── 71_grafana.png │ ├── 71_grafana-home.jpg │ ├── 71_grafana-datasource.jpg │ ├── 71_grafana-manage-dashboard-menu.jpg │ ├── 71_grafana-manage-dashboard-import.jpg │ ├── 71_grafana-manage-dashboard-menu-2.jpg │ ├── 71_grafana-manage-dashboard-success.jpg │ ├── 71_grafana-manage-dashboard-import-done.jpg │ └── 71_grafana-manage-dashboard-import-menu.jpg ├── resources │ └── grafana-metrics.jpg ├── troubleshooting.md ├── intro.md ├── lightning_metrics.md ├── bonus.md └── performance_monitoring.md ├── src ├── Lightning.Metrics │ ├── Lightning.Metrics.csproj.user │ ├── Logger.cs │ ├── Lightning.Metrics.csproj │ ├── MetricConverters │ │ ├── Extensions.cs │ │ ├── LnrpcChannelBalanceResponseMetric.cs │ │ ├── LnrpcWalletBalanceResponseMetric.cs │ │ ├── LnrpcPendingHtlcMetrics.cs │ │ ├── LnrpcNetworkInfoMetric.cs │ │ ├── PendingChannelsResponsePendingOpenChannelMetric.cs │ │ ├── PendingChannelsResponseForceClosedChannelMetrics.cs │ │ └── LnrpcChannelMetrics.cs │ ├── MetricsConfiguration.cs │ ├── NodeAliasCache.cs │ ├── MempoolClient.cs │ └── MetricsClient.cs └── Lightning.Metrics.App │ ├── Lightning.Metrics.App.csproj │ └── Program.cs ├── publish-docker.ps1 ├── Dockerfile ├── arm32.generic.Dockerfile ├── arm32.on.raspberry.Dockerfile ├── LICENSE ├── Lightning.Metrics.sln └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .vs 2 | obj 3 | bin 4 | *.user 5 | launchSetting*.json 6 | -------------------------------------------------------------------------------- /.editorconfig: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/.editorconfig -------------------------------------------------------------------------------- /docs/images/71_grafana.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/images/71_grafana.png -------------------------------------------------------------------------------- /docs/images/71_grafana-home.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/images/71_grafana-home.jpg -------------------------------------------------------------------------------- /docs/resources/grafana-metrics.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/resources/grafana-metrics.jpg -------------------------------------------------------------------------------- /docs/images/71_grafana-datasource.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/images/71_grafana-datasource.jpg -------------------------------------------------------------------------------- /docs/images/71_grafana-manage-dashboard-menu.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/images/71_grafana-manage-dashboard-menu.jpg -------------------------------------------------------------------------------- /docs/images/71_grafana-manage-dashboard-import.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/images/71_grafana-manage-dashboard-import.jpg -------------------------------------------------------------------------------- /docs/images/71_grafana-manage-dashboard-menu-2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/images/71_grafana-manage-dashboard-menu-2.jpg -------------------------------------------------------------------------------- /docs/images/71_grafana-manage-dashboard-success.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/images/71_grafana-manage-dashboard-success.jpg -------------------------------------------------------------------------------- /docs/images/71_grafana-manage-dashboard-import-done.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/images/71_grafana-manage-dashboard-import-done.jpg -------------------------------------------------------------------------------- /docs/images/71_grafana-manage-dashboard-import-menu.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/badokun/lightning-metrics/HEAD/docs/images/71_grafana-manage-dashboard-import-menu.jpg -------------------------------------------------------------------------------- /src/Lightning.Metrics/Lightning.Metrics.csproj.user: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | false 5 | 6 | -------------------------------------------------------------------------------- /publish-docker.ps1: -------------------------------------------------------------------------------- 1 | $ver = [regex]::Match((Get-Content .\src\Lightning.Metrics.App\Lightning.Metrics.App.csproj), '([^<]+)<').Groups[1].Value 2 | git tag -a "v$ver" -m "$ver" 3 | git push --tags 4 | # git tag -d "stable" 5 | # git tag -a "stable" -m "stable" 6 | # git push --tags --force 7 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build 2 | WORKDIR /app 3 | COPY src src/ 4 | WORKDIR /app/src/Lightning.Metrics.App 5 | RUN dotnet publish --runtime linux-x64 -c Release -v minimal -o out 6 | 7 | FROM mcr.microsoft.com/dotnet/runtime:5.0 AS runtime 8 | WORKDIR /app 9 | COPY --from=build /app/src/Lightning.Metrics.App/out ./ 10 | ENTRYPOINT ["/app/lnd-metrics"] 11 | -------------------------------------------------------------------------------- /src/Lightning.Metrics/Logger.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | 3 | namespace Lightning.Metrics 4 | { 5 | public class Logger 6 | { 7 | public static void Debug(string message) 8 | { 9 | Console.WriteLine($"{DateTime.Now:o} DEBUG {message}"); 10 | } 11 | 12 | public static void Error(string message) 13 | { 14 | Console.WriteLine($"{DateTime.Now:o} ERROR {message}"); 15 | } 16 | } 17 | } -------------------------------------------------------------------------------- /src/Lightning.Metrics/Lightning.Metrics.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net5.0 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | -------------------------------------------------------------------------------- /docs/troubleshooting.md: -------------------------------------------------------------------------------- 1 | [ [Intro](intro.md) ] -- [ [Performance Monitoring](performance_monitoring.md) ] -- [ [Lightning Metrics](lightning_metrics.md) ] -- [ [Bonus](bonus.md) ] -- [ [**Troubleshooting**](troubleshooting.md) ] 2 | 3 | ------ 4 | 5 | # Troubleshooting 6 | 7 | Will expand this section as common issues are reported. 8 | 9 | ------ 10 | 11 | Donations 12 | 13 | If you feel like this has been useful and wish to donate, feel free to send a satoshi or two to this address, obviously use Lightning for near free instant transfers: 14 | 15 | * 👉 BTC: `bc1qx2hn38vc8f0fkn3hu8pmpuglg35ctqvx2rzzjs` 16 | * 👉 Lightning: -------------------------------------------------------------------------------- /src/Lightning.Metrics.App/Lightning.Metrics.App.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Exe 5 | net5.0 6 | 1.2.1.0 7 | Lightning.Metrics.App 8 | lnd-metrics 9 | Henno van Rensburg 10 | lnd-metrics 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | -------------------------------------------------------------------------------- /arm32.generic.Dockerfile: -------------------------------------------------------------------------------- 1 | # When targetting a Raspberry Pi 2 | 3 | FROM microsoft/dotnet:2.2-sdk AS build 4 | WORKDIR /app 5 | 6 | # copy csproj and restore as distinct layers 7 | COPY src/Lightning.Metrics.App/*.csproj ./Lightning.Metrics.App/ 8 | COPY src/Lightning.Metrics/*.csproj ./Lightning.Metrics/ 9 | WORKDIR /app/Lightning.Metrics.App 10 | # RUN dotnet restore 11 | RUN dotnet restore -r linux-arm 12 | 13 | # copy and publish app and libraries 14 | WORKDIR /app/ 15 | COPY src/Lightning.Metrics.App/. ./Lightning.Metrics.App/ 16 | COPY src/Lightning.Metrics/. ./Lightning.Metrics/ 17 | WORKDIR /app/Lightning.Metrics.App 18 | RUN dotnet publish -c Release -r linux-arm -o out 19 | 20 | FROM microsoft/dotnet:2.2-runtime-deps-stretch-slim-arm32v7 AS runtime 21 | WORKDIR /app 22 | COPY --from=build /app/Lightning.Metrics.App/out ./ 23 | ENTRYPOINT ["./lnd-metrics"] 24 | -------------------------------------------------------------------------------- /arm32.on.raspberry.Dockerfile: -------------------------------------------------------------------------------- 1 | # When building the image on a Raspberry Pi targetting the same 2 | 3 | FROM microsoft/dotnet:2.2-sdk-stretch-arm32v7 AS build 4 | WORKDIR /app 5 | 6 | # copy csproj and restore as distinct layers 7 | COPY src/Lightning.Metrics.App/*.csproj ./Lightning.Metrics.App/ 8 | COPY src/Lightning.Metrics/*.csproj ./Lightning.Metrics/ 9 | WORKDIR /app/Lightning.Metrics.App 10 | # RUN dotnet restore 11 | RUN dotnet restore -r linux-arm 12 | 13 | # copy and publish app and libraries 14 | WORKDIR /app/ 15 | COPY src/Lightning.Metrics.App/. ./Lightning.Metrics.App/ 16 | COPY src/Lightning.Metrics/. ./Lightning.Metrics/ 17 | WORKDIR /app/Lightning.Metrics.App 18 | RUN dotnet publish -c Release -r linux-arm -o out 19 | 20 | FROM microsoft/dotnet:2.2-runtime-deps-stretch-slim-arm32v7 AS runtime 21 | WORKDIR /app 22 | COPY --from=build /app/Lightning.Metrics.App/out ./ 23 | ENTRYPOINT ["./lnd-metrics"] 24 | -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricConverters/Extensions.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | 3 | namespace Lightning.Metrics.MetricConverters 4 | { 5 | public static class Extensions 6 | { 7 | public const int TagSize = 5; 8 | 9 | public static long ToLong(this string value) 10 | { 11 | return value != null ? long.Parse(value) : 0; 12 | } 13 | 14 | public static int ToInt(this bool? value) 15 | { 16 | return value.HasValue ? Convert.ToInt32(value.Value) : 0; 17 | } 18 | 19 | public static string Left(this string value, int maxLength) 20 | { 21 | if (string.IsNullOrEmpty(value)) 22 | { 23 | return value; 24 | } 25 | 26 | maxLength = Math.Abs(maxLength); 27 | 28 | return (value.Length <= maxLength 29 | ? value 30 | : value.Substring(0, maxLength) 31 | ); 32 | } 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Henno van Rensburg 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricConverters/LnrpcChannelBalanceResponseMetric.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using BTCPayServer.Lightning.LND; 3 | using InfluxDB.Collector; 4 | 5 | namespace Lightning.Metrics.MetricConverters 6 | { 7 | public class LnrpcChannelBalanceResponseMetric 8 | { 9 | private readonly MetricsConfiguration configuration; 10 | private readonly MetricsCollector metrics; 11 | 12 | public LnrpcChannelBalanceResponseMetric(MetricsConfiguration configuration, MetricsCollector metrics) 13 | { 14 | this.configuration = configuration; 15 | this.metrics = metrics; 16 | } 17 | 18 | public void WriteMetrics(LnrpcChannelBalanceResponse channelBalance) 19 | { 20 | this.metrics.Write($"{this.configuration.MetricPrefix}_channel_balance", GetFields(channelBalance)); 21 | } 22 | 23 | private static Dictionary GetFields(LnrpcChannelBalanceResponse channelBalance) 24 | { 25 | return new Dictionary 26 | { 27 | { nameof(channelBalance.Balance).ToLowerInvariant(), channelBalance.Balance.ToLong() }, 28 | { nameof(channelBalance.Pending_open_balance).ToLowerInvariant(), channelBalance.Pending_open_balance.ToLong() } 29 | }; 30 | } 31 | } 32 | } -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricsConfiguration.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | 3 | namespace Lightning.Metrics 4 | { 5 | public enum Network 6 | { 7 | TestNet, 8 | MainNet 9 | } 10 | 11 | public class MetricsConfiguration 12 | { 13 | public Uri InfluxDbUri { get; set; } 14 | public string InfluxDbName { get; set; } 15 | 16 | public Network Network { get; set; } 17 | 18 | public int IntervalSeconds { get; set; } 19 | 20 | public Uri LndRestApiUri { get; set; } 21 | public string MetricPrefix { get; set; } 22 | public string MacaroonHex { get; set; } 23 | public string CertThumbprintHex { get; set; } 24 | 25 | public bool UseMempoolBackend { get; set; } 26 | public string MempoolApiUri { get; set; } 27 | 28 | public void Validate() 29 | { 30 | const int minInterval = 10; 31 | 32 | if (this.IntervalSeconds < minInterval) 33 | { 34 | throw new ArgumentException($"The {nameof(this.IntervalSeconds)} should be greater than {minInterval}"); 35 | } 36 | 37 | if (this.UseMempoolBackend && string.IsNullOrEmpty(this.MempoolApiUri)) 38 | { 39 | throw new ArgumentException($"The {nameof(this.MempoolApiUri)} must not be null if the mempool backend is enabled"); 40 | } 41 | } 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricConverters/LnrpcWalletBalanceResponseMetric.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using BTCPayServer.Lightning.LND; 3 | using InfluxDB.Collector; 4 | 5 | namespace Lightning.Metrics.MetricConverters 6 | { 7 | public class LnrpcWalletBalanceResponseMetric 8 | { 9 | private readonly MetricsConfiguration configuration; 10 | private readonly MetricsCollector metrics; 11 | 12 | public LnrpcWalletBalanceResponseMetric(MetricsConfiguration configuration, MetricsCollector metrics) 13 | { 14 | this.configuration = configuration; 15 | this.metrics = metrics; 16 | } 17 | 18 | public void WriteMetrics(LnrpcWalletBalanceResponse balance) 19 | { 20 | this.metrics.Write($"{this.configuration.MetricPrefix}_balance", GetFields(balance)); 21 | } 22 | 23 | private static Dictionary GetFields(LnrpcWalletBalanceResponse balance) 24 | { 25 | return new Dictionary 26 | { 27 | { nameof(balance.Confirmed_balance).ToLowerInvariant(), balance.Confirmed_balance.ToLong() }, 28 | { nameof(balance.Total_balance).ToLowerInvariant(), balance.Total_balance.ToLong() }, 29 | { nameof(balance.Unconfirmed_balance).ToLowerInvariant(), balance.Unconfirmed_balance.ToLong() } 30 | }; 31 | } 32 | } 33 | } -------------------------------------------------------------------------------- /docs/intro.md: -------------------------------------------------------------------------------- 1 | [ [**Intro**](intro.md) ] -- [ [Performance Monitoring](performance_monitoring.md) ] -- [ [Lightning Metrics](lightning_metrics.md) ] -- [ [Bonus](bonus.md) ] -- [ [Troubleshooting](troubleshooting.md) ] 2 | 3 | ------ 4 | 5 | # Introduction 6 | 7 | This guide assume you have setup your Raspberry using [RaspiBolt Guide](https://github.com/Stadicus/guides/blob/master/raspibolt/README.md) but may work for other configurations as well with some modifications. 8 | 9 | When running Bitcoin and Lightning on a Raspberry it is useful to have more insights into its operation especially since it's a fairly low powered device to begin with. 10 | 11 | Furthermore having historical data available could indicate if a software upgrade to any of the components is having an adverse impact. 12 | 13 | This may help in debugging all sorts of potential problems, e.g. network latency, CPU performance, hard disk free space etc, or even indicate if you are being targeted by hackers. 14 | 15 | This guide has been setup into 3 sections 16 | 17 | ## [Performance Monitoring](performance_monitoring.md) 18 | 19 | Configure recording of system metrics and have it displayed on a website 20 | 21 | ## [Lightning Metrics](lightning_metrics.md) 22 | 23 | Add Lightning specific metrics, e.g. channel size, network capacity 24 | 25 | ## [Bonus](bonus.md) 26 | 27 | Additional activities to make your Raspberry even more awesome 28 | 29 | ------ 30 | 31 | Donations 32 | 33 | If you feel like this has been useful and wish to donate, feel free to send a satoshi or two to this address, obviously use Lightning for near free instant transfers: 34 | 35 | * 👉 BTC: `bc1qx2hn38vc8f0fkn3hu8pmpuglg35ctqvx2rzzjs` 36 | * 👉 Lightning: -------------------------------------------------------------------------------- /src/Lightning.Metrics/NodeAliasCache.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using System.Linq; 3 | using System.Threading.Tasks; 4 | using BTCPayServer.Lightning.LND; 5 | 6 | namespace Lightning.Metrics 7 | { 8 | public class NodeAliasCache 9 | { 10 | private readonly LndClient client; 11 | private readonly Dictionary nodeAliasCache; 12 | 13 | 14 | public NodeAliasCache(LndClient client) 15 | { 16 | this.client = client; 17 | this.nodeAliasCache = new Dictionary(); 18 | } 19 | 20 | public async Task RefreshOnlyIfNecessary(LnrpcListChannelsResponse listChannelsResponse, LnrpcPendingChannelsResponse pendingChannelsResponse) 21 | { 22 | var allPublicKeys = new List(); 23 | 24 | if (listChannelsResponse?.Channels != null) 25 | { 26 | allPublicKeys.AddRange(listChannelsResponse.Channels.Select(c => c.Remote_pubkey)); 27 | } 28 | 29 | if (pendingChannelsResponse?.Pending_open_channels != null) 30 | { 31 | allPublicKeys.AddRange(pendingChannelsResponse.Pending_open_channels.Select(c => c.Channel.Remote_node_pub)); 32 | } 33 | 34 | foreach (var publicKey in allPublicKeys) 35 | { 36 | if (this.nodeAliasCache.ContainsKey(publicKey)) 37 | { 38 | continue; 39 | } 40 | else 41 | { 42 | var nodeInfo = await this.client.SwaggerClient.GetNodeInfoAsync(publicKey).ConfigureAwait(false); 43 | this.nodeAliasCache.Add(nodeInfo.Node.Pub_key, nodeInfo.Node.Alias); 44 | } 45 | } 46 | } 47 | 48 | public string GetNodeAlias(string publicKey) 49 | { 50 | return this.nodeAliasCache[publicKey]; 51 | } 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricConverters/LnrpcPendingHtlcMetrics.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using BTCPayServer.Lightning.LND; 3 | using InfluxDB.Collector; 4 | 5 | namespace Lightning.Metrics.MetricConverters 6 | { 7 | public class LnrpcPendingHtlcMetrics 8 | { 9 | private const string MetricName = "pending_htlcs"; 10 | 11 | private readonly MetricsConfiguration configuration; 12 | private readonly MetricsCollector metrics; 13 | 14 | public LnrpcPendingHtlcMetrics(MetricsConfiguration configuration, MetricsCollector metrics) 15 | { 16 | this.configuration = configuration; 17 | this.metrics = metrics; 18 | } 19 | 20 | public void WriteMetrics(PendingChannelsResponseForceClosedChannel parent) 21 | { 22 | foreach (var pendingHtlcs in parent.Pending_htlcs) 23 | { 24 | this.metrics.Write($"{this.configuration.MetricPrefix}_{MetricName}", GetFields(pendingHtlcs), GetTags(parent)); 25 | } 26 | } 27 | 28 | private static Dictionary GetFields(LnrpcPendingHTLC metric) 29 | { 30 | return new Dictionary 31 | { 32 | { nameof(metric.Amount).ToLowerInvariant(), metric.Amount.ToLong()}, 33 | { nameof(metric.Stage).ToLowerInvariant(), metric.Stage ?? 0}, 34 | { nameof(metric.Outpoint).ToLowerInvariant(), metric.Outpoint.ToLong() }, 35 | { nameof(metric.Blocks_til_maturity).ToLowerInvariant(), metric.Blocks_til_maturity ?? 0 }, 36 | { nameof(metric.Maturity_height).ToLowerInvariant(), metric.Maturity_height ??0 }, 37 | }; 38 | } 39 | 40 | private static Dictionary GetTags(PendingChannelsResponseForceClosedChannel parent) 41 | { 42 | return new Dictionary 43 | { 44 | { nameof(parent.Closing_txid).ToLowerInvariant(), parent.Closing_txid.Left(Extensions.TagSize) } 45 | }; 46 | } 47 | } 48 | } 49 | -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricConverters/LnrpcNetworkInfoMetric.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using BTCPayServer.Lightning.LND; 3 | using InfluxDB.Collector; 4 | 5 | namespace Lightning.Metrics.MetricConverters 6 | { 7 | public class LnrpcNetworkInfoMetric 8 | { 9 | private readonly MetricsConfiguration configuration; 10 | private readonly MetricsCollector metrics; 11 | 12 | public LnrpcNetworkInfoMetric(MetricsConfiguration configuration, MetricsCollector metrics) 13 | { 14 | this.configuration = configuration; 15 | this.metrics = metrics; 16 | } 17 | 18 | public void WriteMetrics(LnrpcNetworkInfo networkInfo) 19 | { 20 | if (networkInfo != null) 21 | { 22 | this.metrics.Write($"{this.configuration.MetricPrefix}_networkinfo", GetFields(networkInfo)); 23 | } 24 | } 25 | 26 | private static Dictionary GetFields(LnrpcNetworkInfo networkInfo) 27 | { 28 | return new Dictionary 29 | { 30 | { nameof(networkInfo.Max_channel_size).ToLowerInvariant(), networkInfo.Max_channel_size.ToLong() }, 31 | { nameof(networkInfo.Min_channel_size).ToLowerInvariant(), networkInfo.Min_channel_size.ToLong() }, 32 | { nameof(networkInfo.Total_network_capacity).ToLowerInvariant(), networkInfo.Total_network_capacity.ToLong()}, 33 | 34 | { nameof(networkInfo.Avg_channel_size).ToLowerInvariant(), networkInfo.Avg_channel_size ?? 0 }, 35 | { nameof(networkInfo.Avg_out_degree).ToLowerInvariant(), networkInfo.Avg_out_degree ?? 0 }, 36 | { nameof(networkInfo.Num_channels).ToLowerInvariant(), networkInfo.Num_channels ?? 0 }, 37 | { nameof(networkInfo.Num_nodes).ToLowerInvariant(), networkInfo.Num_nodes ?? 0 }, 38 | { nameof(networkInfo.Graph_diameter).ToLowerInvariant(), networkInfo.Graph_diameter ?? 0 }, 39 | { nameof(networkInfo.Max_out_degree).ToLowerInvariant(), networkInfo.Max_out_degree ?? 0 } 40 | }; 41 | } 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /Lightning.Metrics.sln: -------------------------------------------------------------------------------- 1 | 2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 16 4 | VisualStudioVersion = 16.0.30804.86 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Lightning.Metrics", "src\Lightning.Metrics\Lightning.Metrics.csproj", "{E87ACE51-4981-4CBE-98A5-BBB529CFEC50}" 7 | EndProject 8 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Lightning.Metrics.App", "src\Lightning.Metrics.App\Lightning.Metrics.App.csproj", "{AA698DC3-9082-462E-9689-14E3A14DE47A}" 9 | EndProject 10 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Resources", "Resources", "{207A500B-1A1F-4CC1-BF4D-34F7B6B00F41}" 11 | ProjectSection(SolutionItems) = preProject 12 | arm32.generic.Dockerfile = arm32.generic.Dockerfile 13 | arm32.on.raspberry.Dockerfile = arm32.on.raspberry.Dockerfile 14 | Dockerfile = Dockerfile 15 | publish-docker.ps1 = publish-docker.ps1 16 | README.md = README.md 17 | EndProjectSection 18 | EndProject 19 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{E495520C-EDF1-4316-A943-71AB154FA4B4}" 20 | ProjectSection(SolutionItems) = preProject 21 | .editorconfig = .editorconfig 22 | EndProjectSection 23 | EndProject 24 | Global 25 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 26 | Debug|Any CPU = Debug|Any CPU 27 | Release|Any CPU = Release|Any CPU 28 | EndGlobalSection 29 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 30 | {E87ACE51-4981-4CBE-98A5-BBB529CFEC50}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 31 | {E87ACE51-4981-4CBE-98A5-BBB529CFEC50}.Debug|Any CPU.Build.0 = Debug|Any CPU 32 | {E87ACE51-4981-4CBE-98A5-BBB529CFEC50}.Release|Any CPU.ActiveCfg = Release|Any CPU 33 | {E87ACE51-4981-4CBE-98A5-BBB529CFEC50}.Release|Any CPU.Build.0 = Release|Any CPU 34 | {AA698DC3-9082-462E-9689-14E3A14DE47A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 35 | {AA698DC3-9082-462E-9689-14E3A14DE47A}.Debug|Any CPU.Build.0 = Debug|Any CPU 36 | {AA698DC3-9082-462E-9689-14E3A14DE47A}.Release|Any CPU.ActiveCfg = Release|Any CPU 37 | {AA698DC3-9082-462E-9689-14E3A14DE47A}.Release|Any CPU.Build.0 = Release|Any CPU 38 | EndGlobalSection 39 | GlobalSection(SolutionProperties) = preSolution 40 | HideSolutionNode = FALSE 41 | EndGlobalSection 42 | GlobalSection(ExtensibilityGlobals) = postSolution 43 | SolutionGuid = {ED6D20A5-4380-462B-B495-7EC527213E33} 44 | EndGlobalSection 45 | EndGlobal 46 | -------------------------------------------------------------------------------- /src/Lightning.Metrics/MempoolClient.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Threading; 4 | using System.Threading.Tasks; 5 | using InfluxDB.Collector; 6 | using RestSharp; 7 | 8 | namespace Lightning.Metrics 9 | { 10 | public class MempoolClient 11 | { 12 | private readonly MetricsConfiguration configuration; 13 | private readonly MetricsCollector metrics; 14 | 15 | private readonly RestClient client; 16 | private readonly RestRequest feesRequest; 17 | 18 | private IRestResponse feesResponse; 19 | 20 | public MempoolClient(MetricsConfiguration configuration, MetricsCollector metrics) 21 | { 22 | this.configuration = configuration; 23 | this.metrics = metrics; 24 | 25 | this.client = new RestClient(configuration.MempoolApiUri) { Timeout = 8000 }; 26 | this.feesRequest = new RestRequest("fees/recommended", DataFormat.Json); 27 | } 28 | 29 | public async Task RequestFeesAsync(CancellationToken ct) 30 | { 31 | this.feesResponse = await this.client.ExecuteAsync(this.feesRequest, ct).ConfigureAwait(false); 32 | } 33 | 34 | public void WriteMetrics() 35 | { 36 | if (this.feesResponse.IsSuccessful) 37 | { 38 | this.metrics.Write($"{this.configuration.MetricPrefix}_recommended_onchain_fees", GetFields(this.feesResponse.Data)); 39 | } 40 | else 41 | { 42 | Logger.Error($"No data could be retrieved from the mempool backend: {this.configuration.MempoolApiUri} ResponseStatus: {this.feesResponse.ResponseStatus}"); 43 | } 44 | } 45 | 46 | private static Dictionary GetFields(RecommendedFees fees) 47 | { 48 | return new Dictionary 49 | { 50 | { nameof(fees.FastestFee).ToLowerInvariant(), fees.FastestFee }, 51 | { nameof(fees.HalfHourFee).ToLowerInvariant(), fees.HalfHourFee }, 52 | { nameof(fees.HourFee).ToLowerInvariant(), fees.HourFee } 53 | }; 54 | } 55 | } 56 | 57 | public class RecommendedFees 58 | { 59 | public int FastestFee { get; set; } 60 | public int HalfHourFee { get; set; } 61 | public int HourFee { get; set; } 62 | } 63 | } 64 | -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricConverters/PendingChannelsResponsePendingOpenChannelMetric.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using BTCPayServer.Lightning.LND; 3 | using InfluxDB.Collector; 4 | 5 | namespace Lightning.Metrics.MetricConverters 6 | { 7 | public class PendingChannelsResponsePendingOpenChannelMetric 8 | { 9 | private readonly MetricsConfiguration configuration; 10 | private readonly MetricsCollector metrics; 11 | private readonly NodeAliasCache nodeAliasCache; 12 | 13 | public PendingChannelsResponsePendingOpenChannelMetric(MetricsConfiguration configuration, MetricsCollector metrics, NodeAliasCache nodeAliasCache) 14 | { 15 | this.configuration = configuration; 16 | this.metrics = metrics; 17 | this.nodeAliasCache = nodeAliasCache; 18 | } 19 | 20 | public void WriteMetrics(LnrpcPendingChannelsResponse pendingChannelsResponse) 21 | { 22 | if (pendingChannelsResponse.Pending_open_channels != null) 23 | { 24 | foreach (var pendingOpen in pendingChannelsResponse.Pending_open_channels) 25 | { 26 | var nodeAlias = this.nodeAliasCache.GetNodeAlias(pendingOpen.Channel.Remote_node_pub); 27 | this.metrics.Write($"{this.configuration.MetricPrefix}_pending_open_channels", GetFields(pendingOpen), GetTags(nodeAlias)); 28 | } 29 | } 30 | } 31 | 32 | private static Dictionary GetFields(PendingChannelsResponsePendingOpenChannel pendingOpenChannel) 33 | { 34 | return new Dictionary 35 | { 36 | { nameof(pendingOpenChannel.Channel.Capacity).ToLowerInvariant(), pendingOpenChannel.Channel.Capacity.ToLong()}, 37 | { nameof(pendingOpenChannel.Channel.Remote_balance).ToLowerInvariant(), pendingOpenChannel.Channel.Remote_balance.ToLong() }, 38 | { nameof(pendingOpenChannel.Channel.Local_balance).ToLowerInvariant(), pendingOpenChannel.Channel.Local_balance.ToLong() }, 39 | { nameof(pendingOpenChannel.Commit_fee).ToLowerInvariant(), pendingOpenChannel.Commit_fee.ToLong() }, 40 | { nameof(pendingOpenChannel.Commit_weight).ToLowerInvariant(), pendingOpenChannel.Commit_weight.ToLong() }, 41 | { nameof(pendingOpenChannel.Fee_per_kw).ToLowerInvariant(), pendingOpenChannel.Fee_per_kw.ToLong() } 42 | }; 43 | } 44 | 45 | private static Dictionary GetTags(string nodeAlias) 46 | { 47 | return new Dictionary 48 | { 49 | { "remote_alias", nodeAlias } 50 | }; 51 | } 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricConverters/PendingChannelsResponseForceClosedChannelMetrics.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using BTCPayServer.Lightning.LND; 3 | using InfluxDB.Collector; 4 | 5 | namespace Lightning.Metrics.MetricConverters 6 | { 7 | public class PendingChannelsResponseForceClosedChannelMetrics 8 | { 9 | private readonly MetricsConfiguration configuration; 10 | private readonly MetricsCollector metrics; 11 | 12 | public PendingChannelsResponseForceClosedChannelMetrics(MetricsConfiguration configuration, MetricsCollector metrics) 13 | { 14 | this.configuration = configuration; 15 | this.metrics = metrics; 16 | } 17 | 18 | public void WriteMetrics(LnrpcPendingChannelsResponse pendingChannelsResponse) 19 | { 20 | if (pendingChannelsResponse.Pending_force_closing_channels != null) 21 | { 22 | foreach (var pendingForceCLose in pendingChannelsResponse.Pending_force_closing_channels) 23 | { 24 | this.metrics.Write($"{this.configuration.MetricPrefix}_forced_closed_channels", GetFields(pendingForceCLose), GetTags(pendingForceCLose)); 25 | 26 | new LnrpcPendingHtlcMetrics(this.configuration, this.metrics).WriteMetrics(pendingForceCLose); 27 | } 28 | } 29 | } 30 | 31 | private static Dictionary GetFields(PendingChannelsResponseForceClosedChannel metric) 32 | { 33 | return new Dictionary 34 | { 35 | { nameof(metric.Channel.Capacity).ToLowerInvariant(), metric.Channel.Capacity.ToLong()}, 36 | { nameof(metric.Channel.Remote_balance).ToLowerInvariant(), metric.Channel.Remote_balance.ToLong() }, 37 | { nameof(metric.Channel.Local_balance).ToLowerInvariant(), metric.Channel.Local_balance.ToLong() }, 38 | 39 | { nameof(metric.Blocks_til_maturity).ToLowerInvariant(), metric.Blocks_til_maturity ?? 0 }, 40 | { nameof(metric.Limbo_balance).ToLowerInvariant(), metric.Limbo_balance.ToLong() }, 41 | { nameof(metric.Maturity_height).ToLowerInvariant(), metric.Maturity_height ??0 }, 42 | { nameof(metric.Recovered_balance).ToLowerInvariant(), metric.Recovered_balance.ToLong() } 43 | }; 44 | } 45 | 46 | private static Dictionary GetTags(PendingChannelsResponseForceClosedChannel metric) 47 | { 48 | return new Dictionary 49 | { 50 | { nameof(metric.Channel.Remote_node_pub).ToLowerInvariant(), metric.Channel.Remote_node_pub.Left(Extensions.TagSize) }, 51 | { nameof(metric.Channel.Channel_point).ToLowerInvariant(), metric.Channel.Channel_point.Left(Extensions.TagSize) } 52 | }; 53 | } 54 | } 55 | } -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricConverters/LnrpcChannelMetrics.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | using BTCPayServer.Lightning.LND; 3 | using InfluxDB.Collector; 4 | 5 | namespace Lightning.Metrics.MetricConverters 6 | { 7 | public class LnrpcChannelMetrics 8 | { 9 | private readonly MetricsConfiguration configuration; 10 | private readonly MetricsCollector metrics; 11 | private readonly NodeAliasCache nodeAliasCache; 12 | 13 | public LnrpcChannelMetrics(MetricsConfiguration configuration, MetricsCollector metrics, NodeAliasCache nodeAliasCache) 14 | { 15 | this.configuration = configuration; 16 | this.metrics = metrics; 17 | this.nodeAliasCache = nodeAliasCache; 18 | } 19 | 20 | public void WriteMetrics(LnrpcListChannelsResponse listChannelsResponse) 21 | { 22 | if (listChannelsResponse?.Channels != null) 23 | { 24 | foreach (var channel in listChannelsResponse.Channels) 25 | { 26 | var nodeAlias = this.nodeAliasCache.GetNodeAlias(channel.Remote_pubkey); 27 | this.metrics.Write($"{this.configuration.MetricPrefix}_list_channels", GetFields(channel), GetTags(nodeAlias)); 28 | } 29 | } 30 | } 31 | 32 | private static Dictionary GetFields(LnrpcChannel metric) 33 | { 34 | return new Dictionary 35 | { 36 | { nameof(metric.Active).ToLowerInvariant(), metric.Active.ToInt() }, 37 | { nameof(metric.Capacity).ToLowerInvariant(), metric.Capacity.ToLong() }, 38 | { nameof(metric.Local_balance).ToLowerInvariant(), metric.Local_balance.ToLong() }, 39 | { nameof(metric.Remote_balance).ToLowerInvariant(), metric.Remote_balance.ToLong() }, 40 | { nameof(metric.Unsettled_balance).ToLowerInvariant(), metric.Unsettled_balance.ToLong() }, 41 | { nameof(metric.Total_satoshis_received).ToLowerInvariant(), metric.Total_satoshis_received.ToLong() }, 42 | { nameof(metric.Total_satoshis_sent).ToLowerInvariant(), metric.Total_satoshis_sent.ToLong() }, 43 | { nameof(metric.Commit_fee).ToLowerInvariant(), metric.Commit_fee.ToLong() }, 44 | { nameof(metric.Commit_weight).ToLowerInvariant(), metric.Commit_weight.ToLong() }, 45 | { nameof(metric.Fee_per_kw).ToLowerInvariant(), metric.Fee_per_kw.ToLong() }, 46 | { nameof(metric.Num_updates).ToLowerInvariant(), metric.Num_updates.ToLong() } 47 | }; 48 | } 49 | 50 | private static Dictionary GetTags(string nodeAlias) 51 | { 52 | return new Dictionary 53 | { 54 | { "remote_alias", nodeAlias } 55 | }; 56 | } 57 | } 58 | } 59 | -------------------------------------------------------------------------------- /src/Lightning.Metrics.App/Program.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.ComponentModel.DataAnnotations; 3 | using System.Reflection; 4 | using System.Threading; 5 | using System.Threading.Tasks; 6 | using McMaster.Extensions.CommandLineUtils; 7 | using Newtonsoft.Json; 8 | // ReSharper disable UnassignedGetOnlyAutoProperty 9 | 10 | namespace Lightning.Metrics.App 11 | { 12 | class Program 13 | { 14 | public static async Task Main(string[] args) => await CommandLineApplication.ExecuteAsync(args); 15 | 16 | [Option("--influxDbUri", Description = "The InfluxDb Uri. E.g. http://192.168.1.40:8086")] 17 | [Required] 18 | public Uri InfluxDbUri { get; } 19 | 20 | [Option("--network", Description = "The bitcoin network. TestNet or MainNet")] 21 | [Required] 22 | public Network Network { get; } = Network.TestNet; 23 | 24 | [Option("--lndRestApiUri", Description = "The Lnd Rest Api Uri. E.g https://192.168.1.40:8080")] 25 | [Required] 26 | public Uri LndRestApiUri { get; } 27 | 28 | [Option("--macaroonHex", Description = "The hex string of the admin.macaroon file. See README.md on how to extract this value")] 29 | [Required] 30 | public string MacaroonHex { get; } 31 | 32 | [Option("--certThumbprintHex", Description = "The hex string of the tls.cert. See README.md on how to extract this value")] 33 | [Required] 34 | public string CertThumbprintHex { get; } 35 | 36 | 37 | [Option("--interval", Description = "The interval in seconds to request metrics. Defaults to 10")] 38 | public int IntervalSeconds { get; } = 10; 39 | 40 | [Option("--influxDbName", Description = "The InfluxDb database name. Defaults to telegraf")] 41 | public string InfluxDbName { get; } = "telegraf"; 42 | 43 | [Option("--metricPrefix", Description = "Prefix all metrics pushed into the InfluxDb. Defaults to lightning")] 44 | public string MetricPrefix { get; } = "lightning"; 45 | 46 | [Option("--use-mempool", Description = "https://github.com/mempool/mempool By default it is disabled.")] 47 | public bool UseMempoolBackend { get; } = false; 48 | 49 | [Option("--mempoolApiUri", Description = "The mempool Rest Api Uri. Defaults to https://mempool.space/api/v1")] 50 | public string MempoolApiUri { get; } = "https://mempool.space/api/v1"; 51 | 52 | 53 | [Option("--test-influxDb", Description = "Test connectivity to the InfluxDb")] 54 | public bool TestInfluxDb { get; } 55 | 56 | [Option("--test-lndApi", Description = "Test connectivity to the Lnd Rest Api")] 57 | public bool TestLndApi { get; } 58 | 59 | private async Task OnExecuteAsync(CancellationToken ct) 60 | { 61 | MetricsConfiguration config = null; 62 | try 63 | { 64 | config = new MetricsConfiguration() 65 | { 66 | InfluxDbUri = InfluxDbUri, 67 | Network = Network, 68 | LndRestApiUri = LndRestApiUri, 69 | MacaroonHex = MacaroonHex, 70 | CertThumbprintHex = CertThumbprintHex, 71 | IntervalSeconds = IntervalSeconds, 72 | InfluxDbName = InfluxDbName, 73 | MetricPrefix = MetricPrefix, 74 | UseMempoolBackend = UseMempoolBackend, 75 | MempoolApiUri = MempoolApiUri 76 | }; 77 | 78 | config.Validate(); 79 | } 80 | catch (Exception e) 81 | { 82 | Console.WriteLine(e.Message); 83 | Environment.Exit(1); 84 | } 85 | 86 | try 87 | { 88 | var client = new MetricsClient(config); 89 | if (this.TestInfluxDb) 90 | { 91 | client.TestInfluxDb(); 92 | } 93 | else if (this.TestLndApi) 94 | { 95 | client.TestLndApi(); 96 | } 97 | else 98 | { 99 | var version = Assembly.GetEntryAssembly().GetCustomAttribute().InformationalVersion; 100 | await client.Start(version, ct).ConfigureAwait(false); 101 | } 102 | } 103 | catch (Exception e) 104 | { 105 | Console.WriteLine(e.Message); 106 | Environment.Exit(1); 107 | } 108 | } 109 | } 110 | } 111 | -------------------------------------------------------------------------------- /docs/lightning_metrics.md: -------------------------------------------------------------------------------- 1 | [ [Intro](intro.md) ] -- [ [Performance Monitoring](performance_monitoring.md) ] -- [ [**Lightning Metrics**](lightning_metrics.md) ] -- [ [Bonus](bonus.md) ] -- [ [Troubleshooting](troubleshooting.md) ] 2 | 3 | ------ 4 | 5 | # Lightning Metrics 6 | 7 | Once you've completed the [Performance Monitoring](performance_monitoring.md) section you are now ready to add Lightning metrics. 8 | 9 | > Your Lightning wallet must be unlocked for metrics to be available 10 | 11 | ## Configuration 12 | 13 | Edit the Lightning daemon's configuration. On a RaspiBolt setup you will find the configuration file at `/home/bitcoin/.lnd/lnd.conf` 14 | 15 | ```yml 16 | [Application Options] 17 | tlsextraip=0.0.0.0 18 | restlisten=0.0.0.0:8080 19 | ``` 20 | 21 | > The `tlsextraip` is required if you plan on running the application on different machine to where the [Lightning Network Daemon](https://github.com/lightningnetwork/lnd) ️is running. 22 | > When adding the `tlsextraip` setting you may need to regenerate the tls.cert, tls.key and macaroon files. To test it's all working access the `/v1/getinfo` endpoint, e.g. You should see `{"error":"expected 1 macaroon, got 0","code":2}` as the response. 23 | 24 | ## Installation 25 | 26 | ### Security 27 | 28 | There are two variables that is required when accessing the LND REST API, `certThumbprintHex` and `macaroonHex` 29 | 30 | #### certThumbprintHex - Extracting the certificate thumbprint 31 | 32 | On a Linux machine execute at the location where you certificate files are, e.g. `/home/bitcoin/.lnd` 33 | 34 | ```bash 35 | openssl x509 -noout -fingerprint -sha256 -inform pem -in tls.cert 36 | ``` 37 | 38 | #### macaroonHex - Extracting the admin.macaroon hex string 39 | 40 | On a Linux machine execute at the location where your macaroon files are, e.g. for testnet `/home/bitcoin/.lnd/data/chain/bitcoin/testnet` 41 | 42 | ```bash 43 | xxd -p admin.macaroon | tr -d '\n' && echo " " 44 | ``` 45 | 46 | In the examples below both `certThumbprintHex` and `macaroonHex` have been shortened for brevity. 47 | 48 | ### Test connectivity to the LND REST API 49 | 50 | ```bash 51 | docker run --rm --net host --name lnd-metrics-arm32-test \ 52 | badokun/lnd-metrics:arm32 \ 53 | --influxDbUri http://127.0.0.1:8086 \ 54 | --network testnet \ 55 | --lndRestApiUri https://127.0.0.1:8080 \ 56 | --certThumbprintHex BC:C5... \ 57 | --macaroonHex 402bb... \ 58 | --test-lndApi 59 | ``` 60 | 61 | `2019-01-14T15:36:14.4858644+00:00 DEBUG LndApi test operation completed successfully` 62 | 63 | ### Test connectivity to the InfluxDb 64 | 65 | ```bash 66 | docker run --rm --net host --name lnd-metrics-arm32-test \ 67 | badokun/lnd-metrics:arm32 \ 68 | --influxDbUri http://127.0.0.1:8086 \ 69 | --network testnet \ 70 | --lndRestApiUri https://127.0.0.1:8080 \ 71 | --certThumbprintHex BC:C5... \ 72 | --macaroonHex 402bb... \ 73 | --test-influxDb 74 | ``` 75 | 76 | `2019-01-14T15:37:01.2207211+00:00 DEBUG InfluxDb write operation completed successfully` 77 | 78 | ### Start collecting metrics 79 | 80 | When you've confirmed connectivity to both the LND REST API and InfluxDb you can omit the `--rm` flag. Pro tip: to keep it always running on a restart add `--restart always` 81 | 82 | ```bash 83 | docker run --restart always -d --net host --name lnd-metrics-arm32 \ 84 | badokun/lnd-metrics:arm32 \ 85 | --influxDbUri http://127.0.0.1:8086 \ 86 | --network testnet \ 87 | --lndRestApiUri https://127.0.0.1:8080 \ 88 | --certThumbprintHex BC:C5... \ 89 | --macaroonHex 402bb... 90 | ``` 91 | 92 | ### Upgrading 93 | 94 | When a new lnd-metrics docker image is released perform the following to upgrade 95 | 96 | ```bash 97 | docker pull badokun/lnd-metrics:arm32 98 | docker stop lnd-metrics-arm32 99 | docker rm lnd-metrics-arm32 100 | 101 | docker run --restart always -d --net host --name lnd-metrics-arm32 \ 102 | badokun/lnd-metrics:arm32 \ 103 | --influxDbUri http://127.0.0.1:8086 \ 104 | --network testnet \ 105 | --lndRestApiUri https://127.0.0.1:8080 \ 106 | --certThumbprintHex BC:C5... \ 107 | --macaroonHex 402bb... 108 | ``` 109 | 110 | ## Grafana Dashboard 111 | 112 | Add a new dashboard in Grafana (refer to the [Performance Monitoring](performance_monitoring.md) on how to do this) 113 | by using the id `9663` 114 | 115 | ![Grafana](resources/grafana-metrics.jpg) 116 | 117 | ## Troubleshooting 118 | 119 | ### Inspect the logs 120 | 121 | If no metrics are being sent to InfluxDb you can run the following command to get the logs `docker logs lnd-metrics-arm32` 122 | 123 | If the logs are littered with messages like below, you need to `unlock` your wallet. In some cases you may need to restart your Lnd daemon. 124 | 125 | `2019-01-14T12:29:47.7104245+00:00 ERROR The HTTP status code of the response was not expected (404).` 126 | 127 | ### Restarting Lnd daemon 128 | 129 | `sudo systemctl restart lnd` 130 | 131 | View the service's log file `sudo journalctl -f -u lnd` 132 | 133 | ------ 134 | 135 | Donations 136 | 137 | If you feel like this has been useful and wish to donate, feel free to send a satoshi or two to this address, obviously use Lightning for near free instant transfers: 138 | 139 | * 👉 BTC: `bc1qx2hn38vc8f0fkn3hu8pmpuglg35ctqvx2rzzjs` 140 | * 👉 Lightning: 141 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # lightning-metrics 2 | 3 | ## Overview 4 | 5 | This application will query a Lightning Node ([LND Rest API](https://api.lightning.community/rest/index.html)) and push all metrics into an InfluxDB which can be used as a data source for Grafana Dashboards similar to the popular [Telegraf](https://github.com/influxdata/telegraf) agent. 6 | 7 | The [RaspiBolt](https://github.com/Stadicus/guides/blob/master/raspibolt/README.md) project served as motivation for setting this up. 8 | If you're looking to run this on a Raspberry Pi that was setup using the guide above [click here](docs/intro.md) 9 | 10 | ## Metrics 11 | 12 | * list_channels 13 | * tags: 14 | * host 15 | * remote_alias 16 | * fields: 17 | * active 18 | * capacity 19 | * local_balance 20 | * remote_balance 21 | * unsettled_balance 22 | * total_satoshis_received 23 | * total_satoshis_sent 24 | * commit_fee 25 | * commit_weight 26 | * fee_per_kw 27 | * num_updates 28 | * pending_open_channels 29 | * tags: 30 | * host 31 | * remote_alias 32 | * fields: 33 | * capacity 34 | * local_balance 35 | * remote_balance 36 | * commit_fee 37 | * commit_weight 38 | * fee_per_kw 39 | * forced_closed_channels 40 | * tags: 41 | * host 42 | * remote_node_pub 43 | * fields: 44 | * capacity 45 | * remote_balance 46 | * local_balance 47 | * blocks_til_maturity 48 | * limbo_balance 49 | * maturity_height 50 | * recovered_balance 51 | * pending_htlcs (as child of forced_closed_channels) 52 | * tags: 53 | * closing_txid 54 | * fields: 55 | * Amount 56 | * Stage 57 | * Outpoint 58 | * Blocks_til_maturity 59 | * Maturity_height 60 | * channel_balance 61 | * tags: 62 | * host 63 | * fields: 64 | * balance 65 | * pending_open_balance 66 | * balance 67 | * tags: 68 | * host 69 | * fields: 70 | * confirmed_balance 71 | * total_balance 72 | * unconfirmed_balance 73 | * networkinfo 74 | * tags: 75 | * host 76 | * fields: 77 | * max_channel_size 78 | * min_channel_size 79 | * total_network_capacity 80 | * avg_channel_size 81 | * avg_out_degree 82 | * num_channels 83 | * num_nodes 84 | * graph_diameter 85 | * max_out_degree 86 | * recommended_onchain_fees 87 | * tags: 88 | * host 89 | * fields: 90 | * fastestFee 91 | * halfHourFee 92 | * hourFee 93 | 94 | ## Configuration 95 | 96 | The application is compiled as `lnd-metrics.exe` and uses the [LND Rest API](https://api.lightning.community/rest/index.html) which 97 | requires the following configuration in the `lnd.conf` file. 98 | 99 | ```bash 100 | [Application Options] 101 | tlsextraip=0.0.0.0 102 | restlisten=0.0.0.0:8080 103 | ``` 104 | 105 | The `tlsextraip` is required if you plan on running the application on different machine to where the [Lightning Network Daemon](https://github.com/lightningnetwork/lnd) ️is running. 106 | > When adding the `tlsextraip` setting you may need to regenerate the tls.cert, tls.key and macaroon files. To test it's all working access the `/v1/getinfo` endpoint, e.g. You should see `{"error":"expected 1 macaroon, got 0","code":2}` as the response. 107 | 108 | ## Usage 109 | 110 | > Your Lightning Wallet needs to be unlocked for the LND REST API to return any data. 111 | 112 | ### Command line 113 | 114 | ```bash 115 | lnd-metrics.exe 116 | --influxDbUri http://192.168.1.40:8086 117 | --network testnet 118 | --lndRestApiUri https://192.168.1.40:8080 119 | --certThumbprintHex "long hex string" 120 | --macaroonHex "long hashed string" 121 | ``` 122 | 123 | To view all the options run 124 | 125 | `lnd-metrics.exe --help` 126 | 127 | #### macaroonHex - Extracting the admin.macaroon hex string 128 | 129 | On a Linux machine execute at the location where your macaroon files are, e.g. for testnet `/home/bitcoin/.lnd/data/chain/bitcoin/testnet` 130 | 131 | ```bash 132 | xxd -p admin.macaroon | tr -d '\n' && echo " " 133 | ``` 134 | 135 | #### certThumbprintHex - Extracting the certificate thumbprint 136 | 137 | On a Linux machine execute at the location where you certificate files are, e.g. `/home/bitcoin/.lnd` 138 | 139 | ```bash 140 | openssl x509 -noout -fingerprint -sha256 -inform pem -in tls.cert 141 | ``` 142 | 143 | ### Docker 144 | 145 | #### On Windows or Linux 146 | 147 | `docker run badokun/lnd-metrics:latest --help` 148 | 149 | #### On RaspBerry Pi 150 | 151 | `docker run badokun/lnd-metrics:arm32 --help` 152 | 153 | ## Development 154 | 155 | ### Building Docker Images 156 | 157 | #### Building a Raspberry compatible image on a Windows or Linux machine 158 | 159 | ```bash 160 | docker build -t lnd-metrics:arm32 -f arm32.generic.Dockerfile . 161 | docker tag lnd-metrics:arm32 badokun/lnd-metrics:arm32 162 | docker push badokun/lnd-metrics:arm32 163 | ``` 164 | 165 | #### Building a Raspberry compatible image on a Raspberry 166 | 167 | ```bash 168 | docker build -t lnd-metrics:arm32 -f arm32.on.raspberry.Dockerfile . 169 | docker tag lnd-metrics:arm32 badokun/lnd-metrics:arm32 170 | docker push badokun/lnd-metrics:arm32 171 | ``` 172 | 173 | #### Building a generic image on Windows or Linux 174 | 175 | ```bash 176 | docker build -t lnd-metrics:latest -f Dockerfile . 177 | docker tag lnd-metrics:latest badokun/lnd-metrics:latest 178 | docker push badokun/lnd-metrics:latest 179 | ``` 180 | 181 | ### Release Management 182 | 183 | * Bump the release version in Lightning.Metrics.App.csproj 184 | * Run `powershell ./publish-docker.ps1` which will create git tags and push to GitHub. 185 | * Docker images are automatically built 186 | 187 | ### Testnet 188 | 189 | Get some free testnet bitcoins at 190 | 191 | ## Resources 192 | 193 | * [Lnd Rest Api](https://api.lightning.community/rest/index.html) 194 | * Setting the `tlsextraip` to `0.0.0.0` was [suggested here](https://github.com/lightningnetwork/lnd/issues/1567#issuecomment-437665324) 195 | * Lnd configuration [reference](https://github.com/lightningnetwork/lnd/blob/master/sample-lnd.conf) 196 | 197 | ------ 198 | 199 | Donations 200 | 201 | If you feel like this has been useful and wish to donate, feel free to send a satoshi or two to this address, obviously use Lightning for near free instant transfers: 202 | 203 | * 👉 BTC: `bc1qx2hn38vc8f0fkn3hu8pmpuglg35ctqvx2rzzjs` 204 | * 👉 Lightning: -------------------------------------------------------------------------------- /src/Lightning.Metrics/MetricsClient.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Threading; 4 | using System.Threading.Tasks; 5 | using BTCPayServer.Lightning; 6 | using BTCPayServer.Lightning.LND; 7 | using InfluxDB.Collector; 8 | using Lightning.Metrics.MetricConverters; 9 | 10 | namespace Lightning.Metrics 11 | { 12 | public class MetricsClient 13 | { 14 | private readonly MetricsConfiguration configuration; 15 | 16 | private DateTime lastNetworkInfoPollingTime = DateTime.MinValue; 17 | 18 | public MetricsClient(MetricsConfiguration configuration) 19 | { 20 | this.configuration = configuration; 21 | } 22 | 23 | public async Task Start(string version, CancellationToken ct) 24 | { 25 | Logger.Debug($"Application v.{version} starting"); 26 | Logger.Debug($"LND Api {this.configuration.LndRestApiUri}"); 27 | Logger.Debug($"InfluxDb {this.configuration.InfluxDbUri}"); 28 | Logger.Debug($"Interval {this.configuration.IntervalSeconds} seconds"); 29 | Logger.Debug($"Colleting metrics commencing"); 30 | 31 | var lndClient = this.CreateLndClient(); 32 | var metrics = this.CreateMetricsCollector(); 33 | var mempoolClient = this.CreateMempoolClientIfEnabled(metrics); 34 | 35 | var nodeAliasCache = new NodeAliasCache(lndClient); 36 | var walletResponseConverter = new LnrpcWalletBalanceResponseMetric(this.configuration, metrics); 37 | var channelBalanceConverter = new LnrpcChannelBalanceResponseMetric(this.configuration, metrics); 38 | var networkInfoConverter = new LnrpcNetworkInfoMetric(this.configuration, metrics); 39 | var pendingOpenChannelConverter = new PendingChannelsResponsePendingOpenChannelMetric(this.configuration, metrics, nodeAliasCache); 40 | var pendingForceClosedChannelConverter = new PendingChannelsResponseForceClosedChannelMetrics(this.configuration, metrics); 41 | var channelMetrics = new LnrpcChannelMetrics(this.configuration, metrics, nodeAliasCache); 42 | 43 | while (!ct.IsCancellationRequested) 44 | { 45 | var waitingTask = Task.Delay(TimeSpan.FromSeconds(this.configuration.IntervalSeconds), ct); 46 | var mempoolTask = mempoolClient?.RequestFeesAsync(ct) ?? Task.CompletedTask; 47 | 48 | try 49 | { 50 | var networkInfo = await this.GetNetworkInfoAfterTenfoldWaitingTime(lndClient, ct).ConfigureAwait(false); 51 | var balance = await lndClient.SwaggerClient.WalletBalanceAsync(ct).ConfigureAwait(false); 52 | var channelBalance = await lndClient.SwaggerClient.ChannelBalanceAsync(ct).ConfigureAwait(false); 53 | var pendingChannels = await lndClient.SwaggerClient.PendingChannelsAsync(ct).ConfigureAwait(false); 54 | var channelList = await lndClient.SwaggerClient.ListChannelsAsync(null, null, null, null, ct).ConfigureAwait(false); 55 | 56 | var refreshTask = nodeAliasCache.RefreshOnlyIfNecessary(channelList, pendingChannels); 57 | 58 | await Task.WhenAll(mempoolTask, refreshTask).ConfigureAwait(false); 59 | 60 | walletResponseConverter.WriteMetrics(balance); 61 | channelBalanceConverter.WriteMetrics(channelBalance); 62 | networkInfoConverter.WriteMetrics(networkInfo); 63 | channelMetrics.WriteMetrics(channelList); 64 | pendingOpenChannelConverter.WriteMetrics(pendingChannels); 65 | pendingForceClosedChannelConverter.WriteMetrics(pendingChannels); 66 | mempoolClient?.WriteMetrics(); 67 | } 68 | catch (Exception e) 69 | { 70 | Logger.Error(e.Message); 71 | 72 | if (e.InnerException != null) 73 | { 74 | Logger.Error(e.InnerException.Message); 75 | } 76 | 77 | lndClient = this.CreateLndClient(); 78 | metrics = this.CreateMetricsCollector(); 79 | } 80 | finally 81 | { 82 | await waitingTask.ConfigureAwait(false); 83 | } 84 | } 85 | } 86 | 87 | public void TestInfluxDb() 88 | { 89 | var metrics = this.CreateMetricsCollector(); 90 | metrics.Write($"{this.configuration.MetricPrefix}_influxDbTest", new Dictionary { { "test", "1" } }); 91 | Logger.Debug("InfluxDb write operation completed successfully"); 92 | } 93 | 94 | public void TestLndApi() 95 | { 96 | var client = this.CreateLndClient(); 97 | var balanceTest = client.SwaggerClient.WalletBalanceAsync(); 98 | balanceTest.Wait(); 99 | Logger.Debug("LndApi test operation completed successfully"); 100 | } 101 | 102 | private async Task GetNetworkInfoAfterTenfoldWaitingTime(LndClient lndClient, CancellationToken ct) 103 | { 104 | if ((DateTime.Now - this.lastNetworkInfoPollingTime).TotalMinutes >= this.configuration.IntervalSeconds) 105 | { 106 | this.lastNetworkInfoPollingTime = DateTime.Now; 107 | 108 | return await lndClient.SwaggerClient.GetNetworkInfoAsync(ct).ConfigureAwait(false); 109 | } 110 | 111 | return null; 112 | } 113 | 114 | private LndClient CreateLndClient() 115 | { 116 | if (!LightningConnectionString.TryParse( 117 | $"type=lnd-rest;server={this.configuration.LndRestApiUri};macaroon={this.configuration.MacaroonHex};certthumbprint={this.configuration.CertThumbprintHex}", 118 | false, out var connectionString)) 119 | { 120 | throw new ArgumentException("Unable to contruct the connection string"); 121 | } 122 | 123 | Logger.Debug("LndClient connection string created successfully"); 124 | 125 | return (LndClient)LightningClientFactory.CreateClient( 126 | connectionString, 127 | this.configuration.Network == Network.MainNet ? NBitcoin.Network.Main : NBitcoin.Network.TestNet); 128 | } 129 | 130 | private MetricsCollector CreateMetricsCollector() 131 | { 132 | return new CollectorConfiguration() 133 | .Tag.With("host", Environment.MachineName) 134 | .Batch.AtInterval(TimeSpan.FromSeconds(this.configuration.IntervalSeconds)) 135 | .WriteTo.InfluxDB(this.configuration.InfluxDbUri, this.configuration.InfluxDbName) 136 | .CreateCollector(); 137 | } 138 | 139 | private MempoolClient CreateMempoolClientIfEnabled(MetricsCollector metrics) 140 | { 141 | return this.configuration.UseMempoolBackend 142 | ? new MempoolClient(this.configuration, metrics) 143 | : null; 144 | } 145 | } 146 | } 147 | -------------------------------------------------------------------------------- /docs/bonus.md: -------------------------------------------------------------------------------- 1 | [ [Intro](intro.md) ] -- [ [Performance Monitoring](performance_monitoring.md) ] -- [ [Lightning Metrics](lightning_metrics.md) ] -- [ [**Bonus**](bonus.md) ] -- [ [Troubleshooting](troubleshooting.md) ] 2 | 3 | ------ 4 | 5 | # Accessing Grafana from the internet 6 | 7 | Having Grafana available on your local network is great, but it would be even better if you can monitor your Raspberry from any location. 8 | 9 | To do this you'll need to setup the following: 10 | 11 | - Get a free domain 12 | - Port forwarding 80 and 443 on your home router to your Raspberry 13 | - Install nginx (engine-x) as a reverse proxy. We do this so we can have all our requests run over https and so we have flexibility in adding more websites later, e.g. [Kibana](https://www.elastic.co/products/kibana), or [BTC RPC Explorer](https://github.com/janoside/btc-rpc-explorer) 14 | - Install certbot for managing your https certificate. 15 | - Restart Grafana docker image with added configuration 16 | 17 | ## Getting a free domain 18 | 19 | There are multiple providers offering free domains, one of them is [freenom](https://my.freenom.com), but really any will do. 20 | 21 | Set your domain to point to your home network's external IP address. This can be obtained by going to [www.whatismyip.com](https://www.whatismyip.com/) 22 | 23 | > For the remainder of this guide, replace `www.lndftw.com` with your domain name. 24 | 25 | ## Router Port forwarding 26 | 27 | You will need to route port 80 (http) and 443 (https) to your Raspberry. Refer to the [ [Raspberry Pi](https://github.com/Stadicus/guides/blob/master/raspibolt/raspibolt_20_pi.md) ] section on how to do this 28 | 29 | ## Update Firewall configuration 30 | 31 | In order to let http and https traffic through run the following: 32 | 33 | ```bash 34 | sudo su 35 | ufw allow 80 comment 'allow http to all' 36 | ufw allow 443 comment 'allow https to all' 37 | exit 38 | ``` 39 | 40 | ## Install nginx as reserve proxy 41 | 42 | When internet traffic over port 443 arrives at your Raspberry we need to ensure it's all encrypted. To do this we use nginx's reverse proxy feature. 43 | 44 | ```bash 45 | sudo apt-get update 46 | sudo apt-get install nginx -y 47 | ``` 48 | 49 | Confirm it's running by running the command below 50 | 51 | ```bash 52 | systemctl status nginx.service 53 | ``` 54 | 55 | ## Prepare nginx for https certificate installation 56 | 57 | Create a configuration file (replace the domain name with yours) 58 | 59 | ```bash 60 | sudo nano /etc/nginx/sites-enabled/www.lndftw.com.conf 61 | ``` 62 | 63 | Paste the text and save 64 | 65 | ```conf 66 | server { 67 | listen 80; 68 | server_name www.lndftw.com; 69 | root /var/www/www.lndftw.com; 70 | location ~ /.well-known { 71 | allow all; 72 | } 73 | } 74 | ``` 75 | 76 | Create directory 77 | 78 | ```bash 79 | sudo mkdir /var/www/www.lndftw.com 80 | sudo chown www-data:www-data /var/www/www.lndftw.com 81 | ``` 82 | 83 | Restart nginx 84 | 85 | ```bash 86 | sudo systemctl restart nginx.service 87 | ``` 88 | 89 | ## Install certbot for your https certificate 90 | 91 | ```bash 92 | sudo sed -i "$ a\deb http://ftp.debian.org/debian stretch-backports main" /etc/apt/sources.list 93 | sudo apt-get update 94 | sudo apt-get install certbot -t stretch-backports -y --force-yes 95 | ``` 96 | 97 | ```bash 98 | sudo certbot certonly -a webroot --webroot-path=/var/www/www.lndftw.com -d www.lndftw.com 99 | ``` 100 | 101 | ## Update nginx configuration 102 | 103 | Now that you have Edit the configuration (replacing the domain name with yours) 104 | 105 | ```bash 106 | sudo nano /etc/nginx/sites-enabled/www.lndftw.com.conf 107 | ``` 108 | 109 | Copy the contents below (replacing the domain name with yours and the `proxy_pass` value with the ip address of your Raspberry) 110 | 111 | ```conf 112 | # Redirect HTTP requests to HTTPS 113 | server { 114 | listen 80; 115 | server_name www.lndftw.com; 116 | return 301 https://$host$request_uri; 117 | } 118 | # For ssl 119 | server { 120 | ssl on; 121 | 122 | ssl_certificate /etc/letsencrypt/live/www.lndftw.com/fullchain.pem; 123 | ssl_certificate_key /etc/letsencrypt/live/www.lndftw.com/privkey.pem; 124 | ssl_protocols TLSv1.1 TLSv1.2; 125 | 126 | default_type application/octet-stream; 127 | 128 | listen 443; 129 | server_name www.lndftw.com; 130 | 131 | root /var/www/www.lndftw.com; 132 | 133 | location ~ /.well-known { 134 | allow all; 135 | } 136 | 137 | index index.html index.htm; 138 | location = / { 139 | return 301 /grafana; 140 | } 141 | location /grafana/ { 142 | proxy_pass http://192.168.1.40:3000/; 143 | } 144 | } 145 | ``` 146 | 147 | Restart nginx 148 | 149 | ```bash 150 | sudo systemctl restart nginx.service 151 | ``` 152 | 153 | ### Reconfigure Grafana 154 | 155 | Grafana needs to be reconfigured in order to work when using a reverse proxy. 156 | 157 | Stop and remove the current Grafana container (your settings will remain intact since it's persisted on another volume) 158 | 159 | ```bash 160 | docker stop grafana 161 | docker rm grafana 162 | ``` 163 | 164 | > Update the domain to yours in the command below 165 | 166 | ```bash 167 | docker run \ 168 | -d \ 169 | -e "GF_SECURITY_ADMIN_PASSWORD=PASSWORD_[A]" \ 170 | -e "GF_SERVER_DOMAIN=www.lndftw.com" \ 171 | -e "GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana/" \ 172 | --name grafana \ 173 | -v grafana-storage:/var/lib/grafana \ 174 | --restart always \ 175 | --net=host \ 176 | grafana/grafana:5.4.3 177 | ``` 178 | 179 | ## Test renewal 180 | 181 | Test that you're able to renew the certificate. 182 | 183 | > Your Raspberry needs to have port 80 and 443 open and routed at this stage 184 | 185 | ```bash 186 | sudo certbot renew --dry-run 187 | ``` 188 | 189 | ### Security strengthening with fail2ban 190 | 191 | Currently a work in progress, documenting the steps to follow soon 192 | 193 | Example regex for auth failures 194 | 195 | ```yml 196 | [Definition] 197 | failregex = ^.*"(GET|POST).*" (404|444|403|400) .*$ 198 | ignoreregex = 199 | ``` 200 | 201 | ## Accessing Grafana inside LAN 202 | 203 | My network configuration didn't allow me to access grafana from inside the work so I had to run another instance on port 3001 204 | 205 | ```bash 206 | docker run \ 207 | -d \ 208 | -e "GF_SECURITY_ADMIN_PASSWORD=PASSWORD_[A]" \ 209 | -e "GF_SERVER_HTTP_PORT=3001" \ 210 | --name grafana-local \ 211 | -v grafana-storage:/var/lib/grafana \ 212 | --restart always \ 213 | --net=host \ 214 | grafana/grafana:5.4.3 215 | ``` 216 | 217 | # Continuous Backup 218 | 219 | Backing your RaspiBolt up may come in handy when the SD cards fails, LND/Bitcoin upgrade, or during a back OS upgrade. 220 | 221 | I prefer the disc image backup which helps you do a system restore 222 | - https://pimylifeup.com/backup-raspberry-pi/ 223 | - logrotate http://www.drdobbs.com/logrotate-a-backup-solution/199101578 224 | 225 | Reference material: 226 | 227 | * 228 | 229 | * 230 | 231 | Further useful reading material - 232 | 233 | ------ 234 | 235 | Donations 236 | 237 | If you feel like this has been useful and wish to donate, feel free to send a satoshi or two to this address, obviously use Lightning for near free instant transfers: 238 | 239 | * 👉 BTC: `bc1qx2hn38vc8f0fkn3hu8pmpuglg35ctqvx2rzzjs` 240 | * 👉 Lightning: -------------------------------------------------------------------------------- /docs/performance_monitoring.md: -------------------------------------------------------------------------------- 1 | [ [Intro](intro.md) ] -- [ [**Performance Monitoring**](performance_monitoring.md) ] -- [ [Lightning Metrics](lightning_metrics.md) ] -- [ [Bonus](bonus.md) ] -- [ [Troubleshooting](troubleshooting.md) ] 2 | 3 | ------ 4 | 5 | # Performance Monitoring on a Raspberry 6 | 7 | > Reference: Thanks to Pete Shima's [medium post](https://medium.com/@petey5000/monitoring-your-home-network-with-influxdb-on-raspberry-pi-with-docker-78a23559ffea) that helped greatly in setting this up. 8 | 9 | ## Overview 10 | 11 | There are a four required pieces to get this working: 12 | 13 | - Docker 14 | - InfluxDB 15 | - Telegraf 16 | - Grafana 17 | 18 | ## Docker 19 | 20 | [Docker](https://www.docker.com) is a computer program that performs operating-system-level virtualization, also known as "containerization". It was first released in 2013 and is developed by Docker, Inc. (source: [Wikipedia](https://en.wikipedia.org/wiki/Docker_(software))) 21 | 22 | - Install Docker by executing the official install script. 23 | 24 | ```bash 25 | cd /home/admin/download 26 | curl -fsSL get.docker.com -o get-docker.sh 27 | sudo sh get-docker.sh 28 | ``` 29 | 30 | - Confirm that Docker is installed correctly. 31 | 32 | ```bash 33 | sudo docker --version 34 | Docker version 18.09.0, build 4d60db4 35 | ``` 36 | 37 | - If you're willing to take the security risk as [outlined here](https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface) you can execute `docker` commands without the `sudo` prefix, alternatively include `sudo` before all docker commands that follow in this guide. 38 | 39 | ```bash 40 | sudo usermod -aG docker $USER 41 | ``` 42 | 43 | - Restart your Raspberry Pi for the changes to take effect and connect as user "admin". 44 | 45 | ```bash 46 | sudo shutdown -r now 47 | ``` 48 | 49 | - Now test Docker by running the "Hello world" image. As it is not yet locally available, Docker automatically retrieves it from the [Docker Hub](https://hub.docker.com/), starts it up and executes the container. 50 | 51 | ```bash 52 | docker run hello-world 53 | ``` 54 | 55 | ## InfluxDB 56 | 57 | [InfluxDB](https://www.influxdata.com/) is an open-source time series database (TSDB) developed by InfluxData. It is written in Go and optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics. (source: [Wikipedia](https://en.wikipedia.org/wiki/InfluxDB)) 58 | 59 | - Start the InfluxDB Docker image with auto-restart in the event of a system restart. 60 | 61 | ```bash 62 | docker run -d --name=influxdb --net=host --restart always --volume=/var/influxdb:/data hypriot/rpi-influxdb 63 | ``` 64 | 65 | - Add a retention policy so we don't have to worry about the InfluxDB growing in size beyond 6 months. 66 | 67 | ```bash 68 | docker ps 69 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 70 | b9f31d893601 hypriot/rpi-influxdb "/usr/bin/entry.sh /…" 5 minutes ago Up 5 minutes influxdb 71 | ``` 72 | 73 | - Use the InfluxDB `CONTAINER ID`, in the example above it's `b9f31d893601`, to open the Influx commandline interface. Enter the commands on lines with `>` directly into the CLI, without the `>`. 74 | 75 | ```bash 76 | $ docker exec -it b9f31d893601 /usr/bin/influx 77 | > CREATE DATABASE telegraf 78 | > USE telegraf 79 | Using database telegraf 80 | > CREATE RETENTION POLICY "six_months" ON "telegraf" DURATION 180d REPLICATION 1 DEFAULT 81 | > SHOW RETENTION POLICIES ON "telegraf" 82 | name duration shardGroupDuration replicaN default 83 | ---- -------- ------------------ -------- ------- 84 | autogen 0s 168h0m0s 1 false 85 | six_months 4320h0m0s 168h0m0s 1 true 86 | 87 | > exit 88 | ``` 89 | 90 | ## Telegraf 91 | 92 | [Telegraf](https://docs.influxdata.com/telegraf) is a plugin-driven server agent for collecting & reporting metrics. It has output plugins to send metrics to a variety of other datastores, services, and message queues, including InfluxDB. 93 | 94 | - Download and install the Telegraf package. 95 | 96 | ```bash 97 | cd /home/admin/download 98 | wget https://dl.influxdata.com/telegraf/releases/telegraf_1.9.4-1_armhf.deb 99 | sudo dpkg -i telegraf_1.9.4-1_armhf.deb 100 | rm telegraf_1.9.4-1_armhf.deb 101 | ``` 102 | 103 | - Telegraf is now installed service. Confirm and check if the program has been started successfully. Press `Ctrl-C` to exit. 104 | 105 | ```bash 106 | sudo systemctl status telegraf 107 | ``` 108 | 109 | - Configure Telegraf by downloading this custom [`telegraf.conf`](https://raw.githubusercontent.com/badokun/guides/master/raspibolt/resources/telegraf.conf) so that it publishes the data we can use later in the Grafana dashboard. 110 | 111 | ```bash 112 | cd /etc/telegraf/ 113 | sudo mv telegraf.conf telegraf.conf.bak 114 | sudo wget https://raw.githubusercontent.com/badokun/guides/master/raspibolt/resources/telegraf.conf 115 | sudo systemctl restart telegraf 116 | ``` 117 | 118 | ## Grafana 119 | 120 | [Grafana](https://grafana.com/) is an open source platform for time series analytics and monitoring. 121 | 122 | - Write down a strong password to access Grafana administration features 123 | 124 | ```bash 125 | [ A ] Grafana Admin password 126 | ``` 127 | 128 | - Create persistent storage for your Grafana configurationso, keeping it also during future upgrades. 129 | 130 | ```bash 131 | sudo docker volume create grafana-storage 132 | ``` 133 | 134 | - Run the Grafana's docker image, replacing the `admin` password setting `PASSWORD_[A]` with your password. This will be used when logging into Grafana's UI. Copy / paste all lines at once into your terminal. 135 | 136 | ```bash 137 | docker run \ 138 | -d \ 139 | -e "GF_SECURITY_ADMIN_PASSWORD=PASSWORD_[A]" \ 140 | --name grafana \ 141 | -v grafana-storage:/var/lib/grafana \ 142 | --restart always \ 143 | --net=host \ 144 | grafana/grafana:5.4.3 145 | ``` 146 | 147 | - Confirm Grafana is running as a docker container. 148 | 149 | ```bash 150 | docker ps 151 | ``` 152 | 153 | ```bash 154 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 155 | 3194df6aff01 grafana/grafana:master "/run.sh" About a minute ago Up About a minute grafana 156 | b9f31d893601 hypriot/rpi-influxdb "/usr/bin/entry.sh /…" 30 minutes ago Up 30 minutes influxdb 157 | ``` 158 | 159 | - To access the analytics webpage, we need to modify the firewall configuration to allow incomming connections to port 3000. 160 | > Note the IP address range, yours may be 192.168.1.0/24 or different. 161 | 162 | ```bash 163 | sudo ufw allow from 192.168.1.0/24 to any port 3000 comment 'allow grafana from local LAN' 164 | ``` 165 | 166 | At this point the basic setup is complete and we can start to setup a Grafana Dashboard. Browse to `http://192.168.1.40:3000` in your browser (use the IP address of your RaspiBolt) and log in with `admin` and `PASSWORD_[A]`. 167 | 168 | ![Grafana Home](images/71_grafana-home.jpg) 169 | 170 | ### Add a data source 171 | 172 | Click on "Add data source", then "InfluxDB". Enter `telegraf` into the Database field 173 | 174 | ![Grafana Data Source](images/71_grafana-datasource.jpg) 175 | 176 | ### Add a Dashboard 177 | 178 | - Locate the shortcut to the left of the page and click on Manage 179 | 180 | ![Grafana Dashboard Menu](images/71_grafana-manage-dashboard-menu.jpg) 181 | 182 | - Importing an existing Dashboard 183 | 184 | ![Grafana Dashboard Menu](images/71_grafana-manage-dashboard-import-menu.jpg) 185 | 186 | - Enter the Grafana Dashboard Id of `9653` and click Load 187 | ![Grafana Dashboard Menu](images/71_grafana-manage-dashboard-import.jpg) 188 | 189 | - Select the InfluxDB from the drop down list and click on Import 190 | ![Grafana Dashboard Menu](images/71_grafana-manage-dashboard-import-done.jpg) 191 | 192 | ## You should see the dashboard in all its glory 193 | 194 | ![Grafana Dashboard Menu](images/71_grafana-manage-dashboard-success.jpg) 195 | 196 | Once you've successfully completed this section, you can take things to the next level by following 197 | [Lightning Metrics](lightning_metrics.md) instructions. 198 | 199 | ------ 200 | 201 | Donations 202 | 203 | If you feel like this has been useful and wish to donate, feel free to send a satoshi or two to this address, obviously use Lightning for near free instant transfers: 204 | 205 | * 👉 BTC: `bc1qx2hn38vc8f0fkn3hu8pmpuglg35ctqvx2rzzjs` 206 | * 👉 Lightning: 207 | --------------------------------------------------------------------------------