5 | Changing the number of executors will only affect future instances and existing instances (if any) will remain unaffected.
6 |
7 |
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetNode/configure-entries.jelly:
--------------------------------------------------------------------------------
1 |
2 |
3 | EC2 Instance
4 |
5 |
--------------------------------------------------------------------------------
/src/test/resources/com/amazon/jenkins/ec2fleet/EC2FleetLabelCloud/name-required-configuration-as-code.yml:
--------------------------------------------------------------------------------
1 | jenkins:
2 | clouds:
3 | - eC2FleetLabel:
4 | awsCredentialsId: xx
5 | region: us-east-2
6 | idleMinutes: 33
7 | minSize: 15
8 | maxSize: 90
9 | numExecutors: 12
--------------------------------------------------------------------------------
/CODE_OF_CONDUCT.md:
--------------------------------------------------------------------------------
1 | ## Code of Conduct
2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
4 | opensource-codeofconduct@amazon.com with any additional questions or comments.
--------------------------------------------------------------------------------
/src/test/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/name-required-configuration-as-code.yml:
--------------------------------------------------------------------------------
1 | jenkins:
2 | clouds:
3 | - eC2Fleet:
4 | awsCredentialsId: xx
5 | region: us-east-2
6 | fleet: my-fleet
7 | labelString: myLabel
8 | idleMinutes: 33
9 | minSize: 15
10 | maxSize: 90
11 | numExecutors: 12
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/help-maxTotalUses.html:
--------------------------------------------------------------------------------
1 |
2 | Set a maximum total uses allowed for instances.
3 | After running 'maximum total uses' amount of jobs, the instance would be terminated.
4 | This setting overrides minSize and minSpareSize, if set.
5 | Use '-1' for unlimited uses.
6 |
7 |
8 | Default: -1 (Unlimited)
9 |
--------------------------------------------------------------------------------
/.mvn/extensions.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 | io.jenkins.tools.incrementals
4 | git-changelist-maven-extension
5 | 1.13
6 |
7 |
8 |
--------------------------------------------------------------------------------
/docs/API.md:
--------------------------------------------------------------------------------
1 | # API
2 | The plugin exposes an API endpoint to retrieve information on the configured clouds in JSON format.
3 |
4 | The endpoint is available by making a GET request to `https:///ec2-fleet/stats`
5 |
6 | The response payload will be:
7 | ```
8 | [
9 | {
10 | "numActive": 0,
11 | "fleet": "fleet-1",
12 | "numDesired": 0,
13 | "state": "active",
14 | "label": "linux"
15 | }
16 | ]
17 | ```
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/help-minSpareSize.html:
--------------------------------------------------------------------------------
1 |
2 | Set a minimum spare size allowed.
3 | The number of instances allowed to be idle / ready to pick up incoming jobs without having to wait for instances to start up.
4 | These instances are exempt from 'Max Idle Minutes Before Scaledown' config.
5 | 'Maximum Cluster Size' is respected if 'Minimum Spare Size' exceeds it.
6 |
2 | Set the interval for checking the EC2 clouds status.
3 |
4 | Every check fetches the current state of the cloud from EC2 and scales the size of the cloud up or down.
5 |
6 |
7 | Longer intervals will reduce the number of EC2 API calls, shorter intervals will allow the cloud to faster scale up and down.
8 |
9 |
10 | The default is to check every 10 seconds.
11 |
12 |
13 |
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/help-endpoint.html:
--------------------------------------------------------------------------------
1 | Optional field, only required if you cannot find proper region in regions list.
2 |
3 | Please specify endpoint for service EC2, example:
4 |
Used to set the number of executors based on the Node's memory(GiB)
2 |
3 |
4 |
Number of Executors =
5 |
11 |
12 |
13 |
14 |
For example, if an EC2 Instance with 8GiB of memory is used as a Node, and the Memory(GiB) Per Executor is set to 2, then
15 | the number of executors will be set to 4.
Used to set the number of executors based on the Node's memory(GiB)
2 |
3 |
4 |
Number of Executors =
5 |
11 |
12 |
13 |
14 |
For example, if an EC2 Instance with a vCPU count of 8 is used as a Node, and the vCPU(s) Per Executor is set to 4, then
15 | the number of executors will be set to 2.
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/AbstractEC2FleetCloud.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import hudson.slaves.Cloud;
4 |
5 | public abstract class AbstractEC2FleetCloud extends Cloud {
6 |
7 | protected AbstractEC2FleetCloud(String name) {
8 | super(name);
9 | }
10 |
11 | public abstract boolean isDisableTaskResubmit();
12 |
13 | public abstract int getIdleMinutes();
14 |
15 | public abstract boolean isAlwaysReconnect();
16 |
17 | public abstract boolean hasExcessCapacity();
18 |
19 | public abstract boolean scheduleToTerminate(String instanceId, boolean ignoreMinConstraints, EC2AgentTerminationReason reason);
20 | }
21 |
--------------------------------------------------------------------------------
/.github/ISSUE_TEMPLATE/feature_request.md:
--------------------------------------------------------------------------------
1 | ---
2 | name: Feature request
3 | about: Suggest an idea for this project
4 | title: ''
5 | labels: enhancement
6 | assignees: ''
7 |
8 | ---
9 |
10 | **Is your feature request related to a problem? Please describe.**
11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
12 |
13 | **Describe the solution you'd like**
14 | A clear and concise description of what you want to happen.
15 |
16 | **Describe alternatives you've considered**
17 | A clear and concise description of any alternative solutions or features you've considered.
18 |
19 | **Additional context**
20 | Add any other context or screenshots about the feature request here.
21 |
--------------------------------------------------------------------------------
/.github/workflows/jenkins-security-scan.yml:
--------------------------------------------------------------------------------
1 | name: Jenkins Security Scan
2 |
3 | on:
4 | push:
5 | branches:
6 | - master
7 | pull_request:
8 | types: [ opened, synchronize, reopened ]
9 | workflow_dispatch:
10 |
11 | permissions:
12 | security-events: write
13 | contents: read
14 | actions: read
15 |
16 | jobs:
17 | security-scan:
18 | uses: jenkins-infra/jenkins-security-scan/.github/workflows/jenkins-security-scan.yaml@v2
19 | with:
20 | java-cache: 'maven' # Optionally enable use of a build dependency cache. Specify 'maven' or 'gradle' as appropriate.
21 | # java-version: 21 # Optionally specify what version of Java to set up for the build, or remove to use a recent default.
22 |
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/JenkinsUtils.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import hudson.model.Node;
4 | import jenkins.model.Jenkins;
5 |
6 | public class JenkinsUtils {
7 |
8 | public static void removeNode(final String instanceId) {
9 | final Jenkins jenkins = Jenkins.get();
10 | // If this node is dying, remove it from Jenkins
11 | final Node n = jenkins.getNode(instanceId);
12 | if (n != null) {
13 | try {
14 | jenkins.removeNode(n);
15 | } catch (final Exception ex) {
16 | throw new IllegalStateException(String.format("Error removing node %s", instanceId), ex);
17 | }
18 | }
19 | }
20 |
21 | }
22 |
--------------------------------------------------------------------------------
/src/test/java/com/amazon/jenkins/ec2fleet/aws/RegionInfoTest.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet.aws;
2 |
3 | import com.amazon.jenkins.ec2fleet.aws.RegionInfo;
4 | import org.junit.jupiter.api.Test;
5 | import software.amazon.awssdk.regions.Region;
6 |
7 | import java.util.List;
8 |
9 | import static org.junit.jupiter.api.Assertions.assertEquals;
10 |
11 | class RegionInfoTest {
12 |
13 | @Test
14 | void verifyRegionInfoDescriptionIsSameAsSDK() {
15 | // Get regions from SDK
16 | final List regions = Region.regions();
17 |
18 | for(final Region region : regions) {
19 | assertEquals(RegionInfo.fromName(region.id()).getDescription(), region.metadata().description());
20 | }
21 | }
22 | }
23 |
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/help-name.html:
--------------------------------------------------------------------------------
1 |
2 | Provide a name for this EC2 Fleet Cloud/EC2 Fleet Label Cloud. If no name is provided, then a default of 'FleetCloud' for a EC2FleetCloud or 'FleetCloudLabel' for a EC2FleetLabelCloud will be used for the name.
3 |
4 |
5 | Could be used in Groovy as cloud identifier.
6 |
7 |
8 |
9 | Note: Due to Jenkins organization,
10 | Build Executor Status section in Jenkins UI could show old cloud name
11 | for agents. To fix that, just wait until agents are recreated (could be long)
12 | or delete the agent (will not delete underlying EC2 instances).
13 | Deleting an agent will cause the plugin to recreate it with correct cloud name.
14 |
7 | Interrupted job will be shown as failed or aborted see pipeline notes below.
8 | While copy will be submitted into jenkins queue to execute or any possible node.
9 |
10 |
11 |
12 | Parametrized job will be resubmitted with same parameters as defined for interrupted.
13 |
14 |
15 |
16 | In case if job is pipeline (workflow) interrupted
17 | task will be aborted as workflow usually don't fail in case of node interruption and wait until
18 | node will be online. Plugin force job to abort and resubmit.
19 |
20 |
21 |
22 | Please report any problem with this feature to GitHub Issue
23 |
24 |
--------------------------------------------------------------------------------
/.github/ISSUE_TEMPLATE/bug_report.md:
--------------------------------------------------------------------------------
1 | ---
2 | name: Bug report
3 | about: Create a bug report to help us improve
4 | title: ''
5 | labels: bug
6 | assignees: ''
7 |
8 | ---
9 |
10 | ## Issue Details
11 |
12 | **Describe the bug**
13 | A clear and concise description of what the bug is.
14 |
15 | **To Reproduce**
16 | 1.
17 | 2.
18 | etc.
19 |
20 | ** Logs **
21 | Tip: See [this guide](https://www.jenkins.io/doc/book/system-administration/viewing-logs/#logs-in-jenkins) to configure a logger in Jenkins UI. Please attach `fine` logs if you think they are relevant.
22 |
23 |
24 | ## Environment Details
25 |
26 | **Plugin Version?**
27 |
28 |
29 | **Jenkins Version?**
30 |
31 |
32 | **Spot Fleet or ASG?**
33 |
34 |
35 | **Label based fleet?**
36 |
37 |
38 | **Linux or Windows?**
39 |
40 |
41 | **EC2Fleet Configuration as Code**
42 | ``
43 | Paste only eC2Fleet part from plugin configuration. Mask all security concerning details. You can download it from Manage Jenkins > Configuration as Code > Download Configuration
44 | ``
45 |
46 | **Anything else unique about your setup?**
47 |
48 |
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/EC2AgentTerminationReason.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | /**
4 | * Enum to represent the reason for termination of an EC2 instance by the plugin.
5 | */
6 | public enum EC2AgentTerminationReason {
7 | IDLE_FOR_TOO_LONG("Agent idle for too long"),
8 | MAX_TOTAL_USES_EXHAUSTED("MaxTotalUses exhausted for agent"),
9 | EXCESS_CAPACITY("Excess capacity for fleet"),
10 | AGENT_DELETED("Agent deleted");
11 |
12 | private final String description;
13 |
14 | EC2AgentTerminationReason(String description) {
15 | this.description = description;
16 | }
17 |
18 | public String getDescription() {
19 | return description;
20 | }
21 |
22 | public static EC2AgentTerminationReason fromDescription(String desc) {
23 | for (EC2AgentTerminationReason reason: values()) {
24 | if(reason.description.equalsIgnoreCase(desc)) {
25 | return reason;
26 | }
27 | }
28 | return null;
29 | }
30 |
31 | @Override
32 | public String toString() {
33 | return this.description;
34 | }
35 | }
36 |
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/Registry.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import com.amazon.jenkins.ec2fleet.aws.CloudFormationApi;
4 | import com.amazon.jenkins.ec2fleet.aws.EC2Api;
5 |
6 | /**
7 | * Decouple plugin code from dependencies for easy testing. We cannot just make transient fields
8 | * in required classes as they usually restored by Jenkins without constructor call. Instead
9 | * we are using registry pattern.
10 | */
11 | @SuppressWarnings("WeakerAccess")
12 | public class Registry {
13 |
14 | private static EC2Api ec2Api = new EC2Api();
15 | private static CloudFormationApi cloudFormationApi = new CloudFormationApi();
16 |
17 | public static void setEc2Api(EC2Api ec2Api) {
18 | Registry.ec2Api = ec2Api;
19 | }
20 |
21 | public static EC2Api getEc2Api() {
22 | return ec2Api;
23 | }
24 |
25 | public static CloudFormationApi getCloudFormationApi() {
26 | return cloudFormationApi;
27 | }
28 |
29 | public static void setCloudFormationApi(CloudFormationApi cloudFormationApi) {
30 | Registry.cloudFormationApi = cloudFormationApi;
31 | }
32 |
33 | }
34 |
--------------------------------------------------------------------------------
/src/test/java/com/amazon/jenkins/ec2fleet/LocalComputerConnector.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import hudson.model.TaskListener;
4 | import hudson.slaves.ComputerConnector;
5 | import hudson.slaves.ComputerLauncher;
6 | import org.jvnet.hudson.test.JenkinsRule;
7 |
8 | import javax.annotation.Nonnull;
9 | import java.io.IOException;
10 | import java.io.Serializable;
11 | import java.net.URISyntaxException;
12 |
13 | /**
14 | * For testing only.
15 | *
16 | * @see AutoResubmitIntegrationTest
17 | */
18 | class LocalComputerConnector extends ComputerConnector implements Serializable {
19 |
20 | @Nonnull
21 | private transient final JenkinsRule j;
22 |
23 | LocalComputerConnector(final JenkinsRule j) {
24 | this.j = j;
25 | }
26 |
27 | @Override
28 | public ComputerLauncher launch(@Nonnull String host, TaskListener listener) throws IOException {
29 | System.out.println("Creating computer launcher");
30 | try {
31 | return j.createComputerLauncher(null);
32 | } catch (URISyntaxException e) {
33 | throw new RuntimeException(e);
34 | }
35 | }
36 | }
37 |
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/EC2ExecutorInterruptionCause.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import jenkins.model.CauseOfInterruption;
4 |
5 | import javax.annotation.Nonnull;
6 |
7 | public class EC2ExecutorInterruptionCause extends CauseOfInterruption {
8 |
9 | @Nonnull
10 | private final String nodeName;
11 |
12 | @SuppressWarnings("WeakerAccess")
13 | public EC2ExecutorInterruptionCause(@Nonnull String nodeName) {
14 | this.nodeName = nodeName;
15 | }
16 |
17 | @Override
18 | public String getShortDescription() {
19 | return "EC2 instance for node " + nodeName + " was terminated";
20 | }
21 |
22 | @Override
23 | public boolean equals(Object o) {
24 | if (o == null || getClass() != o.getClass()) return false;
25 | EC2ExecutorInterruptionCause that = (EC2ExecutorInterruptionCause) o;
26 | return nodeName.equals(that.nodeName);
27 | }
28 |
29 | @Override
30 | public int hashCode() {
31 | return nodeName.hashCode();
32 | }
33 |
34 | @Override
35 | public String toString() {
36 | return getShortDescription();
37 | }
38 |
39 | }
40 |
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/help-noDelayProvision.html:
--------------------------------------------------------------------------------
1 | Enable no delay provision strategy.
2 |
3 | Disabled by default.
4 |
5 |
6 | Enabling this setting will disable default Jenkins behavior for a particular cloud
7 | and ask this cloud to provision new agents almost as soon as new jobs appear in the queue.
8 |
9 |
10 | When this option is disabled (by default):
11 | Each time Jenkins doesn't have enough agents to execute all jobs in a queue,
12 | Jenkins starts provisioning new agents. The default strategy tries to keep the amount of agents
13 | as small as possible. Jenkins waits a little bit until the number of jobs in the queue reaches some limit or
14 | jobs in the queue reach some age. As a result, before a scheduled job will be executed you can
15 | see some delay.
16 |
17 |
18 | Note: Enabling this setting can increase the chance of over-provisioning. Because of that,
19 | make sure that your Max Idle Minutes Before Scaledown is set so that
20 | free capacity can be scaled in. While Linux capacity is billed per minute, Windows capacity
21 | is billed per hour, so be careful with this option.
22 |
23 |
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/auto-scaling-group.yml:
--------------------------------------------------------------------------------
1 | Parameters:
2 | InstanceType:
3 | Type: String
4 | ImageId:
5 | Type: String
6 | SpotPrice:
7 | Type: String
8 | MaxSize:
9 | Type: String
10 | MinSize:
11 | Type: String
12 | KeyName:
13 | Type: String
14 |
15 | Outputs:
16 | FleetId:
17 | Value:
18 | Ref: ASG
19 |
20 | Resources:
21 | # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig.html#aws-properties-as-launchconfig-properties
22 | ASGLaunchConfig:
23 | Type: AWS::AutoScaling::LaunchConfiguration
24 | Properties:
25 | ImageId:
26 | Ref: ImageId
27 | # UserData:
28 | # Fn::Base64:
29 | # Ref: "WebServerPort"
30 | InstanceType:
31 | Ref: InstanceType
32 | SpotPrice:
33 | Ref: SpotPrice
34 | KeyName:
35 | Ref: KeyName
36 |
37 | ASG:
38 | Type: AWS::AutoScaling::AutoScalingGroup
39 | Properties:
40 | AvailabilityZones:
41 | Fn::GetAZs:
42 | Ref: AWS::Region
43 | LaunchConfigurationName:
44 | Ref: ASGLaunchConfig
45 | MinSize:
46 | Ref: MinSize
47 | MaxSize:
48 | Ref: MaxSize
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/fleet/EC2Fleet.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet.fleet;
2 |
3 | import com.amazon.jenkins.ec2fleet.FleetStateStats;
4 | import hudson.util.ListBoxModel;
5 |
6 | import java.util.Collection;
7 | import java.util.Map;
8 |
9 | /**
10 | * Hide details of access to EC2 Fleet depending on implementation like EC2 Fleet, Spot Fleet,
11 | * or Auto Scaling Group.
12 | *
13 | * @see EC2EC2Fleet
14 | * @see EC2SpotFleet
15 | * @see AutoScalingGroupFleet
16 | */
17 | public interface EC2Fleet {
18 |
19 | void describe(
20 | final String awsCredentialsId, final String regionName, final String endpoint,
21 | final ListBoxModel model, final String selectedId, final boolean showAll);
22 |
23 | void modify(
24 | final String awsCredentialsId, final String regionName, final String endpoint,
25 | final String id, final int targetCapacity, int min, int max);
26 |
27 | FleetStateStats getState(
28 | final String awsCredentialsId, final String regionName, final String endpoint,
29 | final String id);
30 |
31 | Map getStateBatch(
32 | final String awsCredentialsId, final String regionName, final String endpoint,
33 | final Collection ids);
34 |
35 | Boolean isAutoScalingGroup();
36 |
37 | }
38 |
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/help-fleet.html:
--------------------------------------------------------------------------------
1 | Select the fleet which will be used to launch Jenkins workers. EC2 Spot Fleet and Auto Scaling Group
2 | are both supported.
3 |
4 | By default, this list only includes active
5 | EC2 Spot Fleets and Auto Scaling Groups. If you need to see all fleets,
6 | click the checkbox below. The selected fleet will be in the list regardless of fleet state.
7 |
8 |
9 | EC2 Spot Fleet specific: All non maintain type fleets will be filtered out from this list.
10 | More
11 | about types.
12 |
13 | If you are not sure about your fleet type, check its type via an API call or look in the console UI for
14 | Maintain Target Capacity being enabled.
15 |
16 |
17 | AWS Permissions: The plugin requires certain AWS permissions for managing resources. You can check missing permissions by clicking the Test Connection button.
18 | The Test Connection button does not validate certain listed permissions such as TerminateInstances and UpdateAutoScalingGroup due to API restrictions.
19 | Check out the README for an inline IAM Policy.
20 |
Controls how Jenkins schedules builds on provisioned nodes.
3 | When unchecked
4 |
5 | Jenkins will use provisioned nodes as much as possible.
6 | This is the default setting. In this mode, Jenkins uses provisioned nodes freely.
7 | Whenever there is a build that can be done by using provisioned node, Jenkins will use it.
8 |
9 |
10 | When checked
11 |
12 | In this mode, Jenkins will only build a project on provisioned node when that project is restricted to certain nodes
13 | using a label expression, and that expression matches this node's labels.
14 |
15 |
16 | This allows a node to be reserved for certain kinds of jobs. For example, if you have jobs that run performance
17 | tests, you may want them to only run on a specially configured machine, while preventing all other jobs from
18 | using that machine. To do so, you would restrict where the test jobs may run by giving them a label expression
19 | matching that machine.
20 |
21 |
22 | Furthermore, if you set the # of executors value to 1, you can ensure that only one performance test will
23 | execute at any given time on that machine; no other builds will interfere.
24 |
25 |
26 |
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/help-initOnlineTimeoutSec.html:
--------------------------------------------------------------------------------
1 | The maximum time Jenkins will wait for an EC2 instance to become active before
2 | trying to request a new instance from the fleet.
3 |
4 |
5 | By default, Jenkins expects that a node will be available immediately. However, EC2 instances
6 | require some time to be provisioned, start, and be ready for builds. Therefore,
7 | Jenkins might request more capacity than is actually required (over-provision). This setting
8 | prevents over-provisioning by forcing Jenkins to wait until an EC2 instance is online before requesting
9 | more capacity.
10 |
11 |
12 |
13 | Cannot be negative.
14 | For positive values, force Jenkins to wait up to the given value for an instance to come online.
15 | If set to 0, the timeout will be disabled and Jenkins will
16 | not wait any time to start up EC2 instance, which could lead to over-provisioning.
17 |
18 |
19 | Note: Be careful with exceedingly large values (more than 5 min) for this setting. If any problem
20 | prevents the node from coming online (no Java, wrong version of Java, Jenkins agent cannot start, etc.),
21 | Jenkins will wait without requesting additional capacity.
22 |
23 |
24 | Behavior of versions of plugin before 1.8.0 could be represented as this version
25 | with timeout set to 0.
26 |
8 | When checked, the plugin consumes instance weight information provided by a Launch Specification
9 | and uses it to scale the node's number of executors from configuration field Number of Executors.
10 | Note: current implementation doesn't support Launch Template, only Launch Specification.
11 |
12 |
13 |
14 | Example (here instance type from launch specification matches with
15 | launched instance type):
16 |
17 |
18 |
19 |
20 |
Number of Executors
21 |
Instance Weight
22 |
Effective Number of Executors
23 |
24 |
25 |
26 |
27 |
1
28 |
1
29 |
1
30 |
31 |
32 |
1
33 |
0.5
34 |
1
35 |
36 |
37 |
1
38 |
0.1
39 |
1
40 |
41 |
42 |
10
43 |
0.1
44 |
1
45 |
46 |
47 |
1
48 |
1.5
49 |
2
50 |
51 |
52 |
1
53 |
1.44
54 |
1
55 |
56 |
57 |
58 |
59 |
60 |
61 | Plugin always set number of executors to at least one.
62 | If the launched instance type doesn't match any weight in launch specification,
63 | regular number of executors will be used without any scaling.
64 |
65 |
--------------------------------------------------------------------------------
/src/test/java/com/amazon/jenkins/ec2fleet/EC2FleetStatsApiTest.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import com.amazon.jenkins.ec2fleet.FleetStateStats;
4 | import com.amazon.jenkins.ec2fleet.EC2FleetCloud;
5 | import org.junit.jupiter.api.BeforeEach;
6 | import org.junit.jupiter.api.Test;
7 | import org.jvnet.hudson.test.JenkinsRule;
8 | import org.jvnet.hudson.test.junit.jupiter.WithJenkins;
9 | import org.mockito.Mockito;
10 |
11 | import java.io.InputStream;
12 | import java.net.URL;
13 | import java.nio.charset.StandardCharsets;
14 | import java.util.Arrays;
15 | import java.util.Collections;
16 | import java.util.HashSet;
17 |
18 | import static org.junit.jupiter.api.Assertions.*;
19 |
20 | @WithJenkins
21 | class EC2FleetStatsApiTest {
22 |
23 | private JenkinsRule j;
24 |
25 | @BeforeEach
26 | void before(JenkinsRule rule) {
27 | j = rule;
28 | }
29 |
30 | @Test
31 | void testFleetStatsApiEndpointReturnsExpectedJson() throws Exception {
32 | FleetStateStats.State state = new FleetStateStats.State(true, false, "active");
33 | FleetStateStats stats = new FleetStateStats("fleet-1", 2, state,
34 | new HashSet<>(Arrays.asList("i-1", "i-2")), Collections.emptyMap());
35 |
36 | EC2FleetCloud cloud = Mockito.mock(EC2FleetCloud.class);
37 | Mockito.when(cloud.getStats()).thenReturn(stats);
38 | Mockito.when(cloud.getFleet()).thenReturn("fleet-1");
39 | Mockito.when(cloud.getLabelString()).thenReturn("label-1");
40 |
41 | j.jenkins.clouds.add(cloud);
42 |
43 | String url = j.getURL() + "ec2-fleet/stats";
44 | try (InputStream is = new URL(url).openStream()) {
45 | String json = new String(is.readAllBytes(), StandardCharsets.UTF_8);
46 | assertTrue(json.contains("\"fleet\":\"fleet-1\""));
47 | assertTrue(json.contains("\"state\":\"active\""));
48 | assertTrue(json.contains("\"label\":\"label-1\""));
49 | assertTrue(json.contains("\"numActive\":2"));
50 | }
51 | }
52 | }
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/CloudNames.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import jenkins.model.Jenkins;
4 | import org.apache.commons.lang.RandomStringUtils;
5 |
6 | import java.util.Collections;
7 | import java.util.HashSet;
8 | import java.util.Set;
9 | import java.util.stream.Collectors;
10 |
11 | public class CloudNames {
12 |
13 | public static final int SUFFIX_LENGTH = 8;
14 |
15 | private static final Set usedSuffixes = new HashSet();
16 |
17 | public static Boolean isUnique(final String name) {
18 | return !Jenkins.get().clouds.stream().anyMatch(c -> c.name.equals(name));
19 | }
20 |
21 | public static Boolean isDuplicated(final String name) { return Jenkins.get().clouds.stream().filter(c -> c.name.equals(name)).count() > 1; }
22 |
23 | public static String generateUnique(final String proposedName) {
24 | final Set usedNames = Jenkins.get().clouds != null
25 | ? Jenkins.get().clouds.stream().map(c -> c.name).collect(Collectors.toSet())
26 | : Collections.emptySet();
27 |
28 | if (proposedName.equals(EC2FleetCloud.BASE_DEFAULT_FLEET_CLOUD_ID) || proposedName.equals(EC2FleetLabelCloud.BASE_DEFAULT_FLEET_CLOUD_ID) || usedNames.contains(proposedName)) {
29 | return proposedName + "-" + generateSuffix();
30 | }
31 |
32 | return proposedName;
33 | }
34 |
35 | /**
36 | * We are using a randomly generated string as a suffix here because Jenkins creates its clouds as a batch.
37 | * This functionality means if a CasC user has two empty strings for the name field prompting two clouds with
38 | * default names, they would theoretically have the same default name. By appending a random suffix we can be
39 | * sure w.h.p that the suffixes created will be different.
40 | */
41 | private static String generateSuffix() {
42 | String suffix;
43 |
44 | do {
45 | suffix = RandomStringUtils.randomAlphanumeric(SUFFIX_LENGTH);
46 | } while (usedSuffixes.contains(suffix));
47 |
48 | usedSuffixes.add(suffix);
49 | return suffix;
50 | }
51 | }
52 |
--------------------------------------------------------------------------------
/.github/workflows/cd.yaml:
--------------------------------------------------------------------------------
1 | # Note: additional setup is required, see https://www.jenkins.io/redirect/continuous-delivery-of-plugins
2 | #
3 | # Please find additional hints for individual trigger use case
4 | # configuration options inline this script below.
5 | #
6 | ---
7 | name: cd
8 | on:
9 | workflow_dispatch:
10 | inputs:
11 | validate_only:
12 | required: false
13 | type: boolean
14 | description: |
15 | Run validation with release drafter only
16 | → Skip the release job
17 | # Note: Change this default to true,
18 | # if the checkbox should be checked by default.
19 | default: false
20 | # If you don't want any automatic trigger in general, then
21 | # the following check_run trigger lines should all be commented.
22 | # Note: Consider the use case #2 config for 'validate_only' below
23 | # as an alternative option!
24 | # check_run:
25 | # types:
26 | # - completed
27 |
28 | permissions:
29 | checks: read
30 | contents: write
31 |
32 | jobs:
33 | maven-cd:
34 | uses: jenkins-infra/github-reusable-workflows/.github/workflows/maven-cd.yml@v1
35 | with:
36 | # Comment / uncomment the validate_only config appropriate to your preference:
37 | #
38 | # Use case #1 (automatic release):
39 | # - Let any successful Jenkins build trigger another release,
40 | # if there are merged pull requests of interest
41 | # - Perform a validation only run with drafting a release note,
42 | # if manually triggered AND inputs.validate_only has been checked.
43 | #
44 | validate_only: ${{ inputs.validate_only == true }}
45 | #
46 | # Alternative use case #2 (no automatic release):
47 | # - Same as use case #1 - but:
48 | # - Let any check_run trigger a validate_only run.
49 | # => enforce the release job to be skipped.
50 | #
51 | #validate_only: ${{ inputs.validate_only == true || github.event_name == 'check_run' }}
52 | secrets:
53 | MAVEN_USERNAME: ${{ secrets.MAVEN_USERNAME }}
54 | MAVEN_TOKEN: ${{ secrets.MAVEN_TOKEN }}
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/EC2FleetStatusInfo.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import hudson.widgets.Widget;
4 |
5 | import javax.annotation.concurrent.ThreadSafe;
6 | import java.util.Objects;
7 |
8 | /**
9 | * This consumed by jelly file EC2FleetStatusWidget/index.jelly
10 | * to render fleet information about all fleets, don't forget to update it
11 | * if you change fields name
12 | *
13 | * @see EC2FleetStatusWidget
14 | * @see CloudNanny
15 | */
16 | @SuppressWarnings({"WeakerAccess", "unused"})
17 | @ThreadSafe
18 | public class EC2FleetStatusInfo extends Widget {
19 |
20 | private final String id;
21 | private final String state;
22 | private final String label;
23 | private final int numActive;
24 | private final int numDesired;
25 |
26 | public EC2FleetStatusInfo(String id, String state, String label, int numActive, int numDesired) {
27 | this.id = id;
28 | this.state = state;
29 | this.label = label;
30 | this.numActive = numActive;
31 | this.numDesired = numDesired;
32 | }
33 |
34 | public String getId() {
35 | return id;
36 | }
37 |
38 | @Override
39 | public boolean equals(Object o) {
40 | if (this == o) return true;
41 | if (o == null || getClass() != o.getClass()) return false;
42 | EC2FleetStatusInfo that = (EC2FleetStatusInfo) o;
43 | return numActive == that.numActive &&
44 | numDesired == that.numDesired &&
45 | Objects.equals(id, that.id) &&
46 | Objects.equals(state, that.state) &&
47 | Objects.equals(label, that.label);
48 | }
49 |
50 | @Override
51 | public int hashCode() {
52 | return Objects.hash(id, state, label, numActive, numDesired);
53 | }
54 |
55 | public String getLabel() {
56 | return label;
57 | }
58 |
59 | public String getState() {
60 | return state;
61 | }
62 |
63 | public int getNumActive() {
64 | return numActive;
65 | }
66 |
67 | public int getNumDesired() {
68 | return numDesired;
69 | }
70 |
71 | }
72 |
--------------------------------------------------------------------------------
/.github/workflows/stale.yml:
--------------------------------------------------------------------------------
1 | name: Stale Issues / PRs
2 |
3 | on:
4 | schedule:
5 | - cron: "0 17 * * *" # Runs every day at 12:00PM CST
6 |
7 | jobs:
8 | stale:
9 | runs-on: ubuntu-latest
10 | steps:
11 | # 15+5 day stale policy for PRs
12 | # * Except PRs marked as "stalebot-ignore"
13 | - name: Stale PRs policy
14 | uses: actions/stale@v10.1.0
15 | with:
16 | repo-token: ${{ secrets.GITHUB_TOKEN }}
17 | exempt-pr-labels: "stalebot-ignore"
18 | days-before-stale: 15
19 | days-before-close: 5
20 | days-before-issue-stale: -1
21 | days-before-issue-close: -1
22 | remove-stale-when-updated: true
23 | stale-pr-label: "stale"
24 | operations-per-run: 100
25 | stale-pr-message: >
26 | This PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
27 | If you want this PR to never become stale, please ask a maintainer to apply the "stalebot-ignore" label.
28 | close-pr-message: >
29 | This PR was closed because it has become stale with no activity.
30 |
31 | # 30+5 day stale policy for open issues
32 | # * Except Issues marked as "stalebot-ignore"
33 | - name: Stale Issues policy
34 | uses: actions/stale@v10.1.0
35 | with:
36 | repo-token: ${{ secrets.GITHUB_TOKEN }}
37 | exempt-issue-labels: "stalebot-ignore"
38 | days-before-stale: 30
39 | days-before-close: 5
40 | days-before-pr-stale: -1
41 | days-before-pr-close: -1
42 | remove-stale-when-updated: true
43 | stale-issue-label: "stale"
44 | operations-per-run: 100
45 | stale-issue-message: >
46 | This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
47 | If you want this issue to never become stale, please ask a maintainer to apply the "stalebot-ignore" label.
48 | close-issue-message: >
49 | This issue was closed because it has become stale with no activity.
50 |
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/help-executorScaler.html:
--------------------------------------------------------------------------------
1 |
Plugin always set number of executors to at least one.
2 |
No Scaling
3 |
Doesn't scale. Uses set number of executors
4 |
5 |
Scale by Node Hardware
6 |
Determine number of executors by node hardware: Memory and vCPU count. Specify the quantity of each resource to be
7 | allocated per executor. The plugin sources the resources of the node and calculates the desired number of executors.
8 | The lower number of executors calculated will be chosen to prevent over provisioning.
9 |
10 |
11 |
Scale by Weight
12 |
13 |
14 | The plugin consumes instance
15 | weight information provided by a Launch Specification
16 | and uses it to scale the node's number of executors from configuration field Number of Executors.
17 | Note: current implementation doesn't support Launch Template, only Launch Specification.
18 |
19 |
20 |
21 | Example (here instance type from launch specification matches with
22 | launched instance type):
23 |
24 |
25 |
26 |
27 |
Number of Executors
28 |
Instance Weight
29 |
Effective Number of Executors
30 |
31 |
32 |
33 |
34 |
1
35 |
1
36 |
1
37 |
38 |
39 |
1
40 |
0.5
41 |
1
42 |
43 |
44 |
1
45 |
0.1
46 |
1
47 |
48 |
49 |
10
50 |
0.1
51 |
1
52 |
53 |
54 |
1
55 |
1.5
56 |
2
57 |
58 |
59 |
1
60 |
1.44
61 |
1
62 |
63 |
64 |
65 |
66 |
67 |
68 | If the launched instance type doesn't match any weight in launch specification,
69 | regular number of executors will be used without any scaling.
70 |
Exceptions
28 | * This method will be executed by {@link PeriodicWork} inside {@link java.util.concurrent.ScheduledExecutorService}
29 | * by default it stops execution if task throws exception, however {@link PeriodicWork} fix that
30 | * by catch any exception and just log it, so we safe to throw exception here.
31 | */
32 | @Override
33 | protected void doRun() {
34 | final List info = new ArrayList<>();
35 | for (final Cloud cloud : getClouds()) {
36 | if (!(cloud instanceof EC2FleetCloud)) continue;
37 | final EC2FleetCloud fleetCloud = (EC2FleetCloud) cloud;
38 | final FleetStateStats stats = fleetCloud.getStats();
39 | // could be when plugin just started and not yet updated, ok to skip
40 | if (stats == null) continue;
41 |
42 | info.add(new EC2FleetStatusInfo(
43 | fleetCloud.getFleet(), stats.getState().getDetailed(), fleetCloud.getLabelString(),
44 | stats.getNumActive(), stats.getNumDesired()));
45 | }
46 |
47 | for (final Widget w : getWidgets()) {
48 | if (w instanceof EC2FleetStatusWidget) ((EC2FleetStatusWidget) w).setStatusList(info);
49 | }
50 | }
51 |
52 | /**
53 | * Will be mocked by tests to avoid deal with jenkins
54 | *
55 | * @return widgets
56 | */
57 | @VisibleForTesting
58 | static List getWidgets() {
59 | return Jenkins.get().getWidgets();
60 | }
61 |
62 | /**
63 | * We return {@link List} instead of original {@link Jenkins.CloudList}
64 | * to simplify testing as jenkins list requires actual {@link Jenkins} instance.
65 | *
66 | * @return basic java list
67 | */
68 | @VisibleForTesting
69 | static List getClouds() {
70 | return Jenkins.get().clouds;
71 | }
72 |
73 | }
74 |
--------------------------------------------------------------------------------
/src/test/java/com/amazon/jenkins/ec2fleet/EC2FleetLabelParametersTest.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import org.junit.jupiter.api.Test;
4 |
5 | import static org.junit.jupiter.api.Assertions.assertEquals;
6 | import static org.junit.jupiter.api.Assertions.assertNull;
7 |
8 | class EC2FleetLabelParametersTest {
9 |
10 | @Test
11 | void parse_emptyForEmptyString() {
12 | final EC2FleetLabelParameters parameters = new EC2FleetLabelParameters("");
13 | assertNull(parameters.get("aa"));
14 | }
15 |
16 | @Test
17 | void parse_emptyForNullString() {
18 | final EC2FleetLabelParameters parameters = new EC2FleetLabelParameters(null);
19 | assertNull(parameters.get("aa"));
20 | }
21 |
22 | @Test
23 | void parse_forString() {
24 | final EC2FleetLabelParameters parameters = new EC2FleetLabelParameters("a=1,b=2");
25 | assertEquals("1", parameters.get("a"));
26 | assertEquals("2", parameters.get("b"));
27 | assertNull(parameters.get("c"));
28 | }
29 |
30 | @Test
31 | void get_caseInsensitive() {
32 | final EC2FleetLabelParameters parameters = new EC2FleetLabelParameters("aBc=1");
33 | assertEquals("1", parameters.get("aBc"));
34 | assertEquals("1", parameters.get("ABC"));
35 | assertEquals("1", parameters.get("abc"));
36 | assertEquals("1", parameters.get("AbC"));
37 | assertEquals("1", parameters.getOrDefault("AbC", "?"));
38 | assertEquals(1, parameters.getIntOrDefault("AbC", -1));
39 | }
40 |
41 | @Test
42 | void parse_withFleetNamePrefixSkipItAndProvideParameters() {
43 | final EC2FleetLabelParameters parameters = new EC2FleetLabelParameters("AA_a=1,b=2");
44 | assertEquals("1", parameters.get("a"));
45 | assertEquals("2", parameters.get("b"));
46 | assertNull(parameters.get("c"));
47 | }
48 |
49 | @Test
50 | void parse_withEmptyFleetNamePrefixSkipItAndProvideParameters() {
51 | final EC2FleetLabelParameters parameters = new EC2FleetLabelParameters("_a=1,b=2");
52 | assertEquals("1", parameters.get("a"));
53 | assertEquals("2", parameters.get("b"));
54 | assertNull(parameters.get("c"));
55 | }
56 |
57 | @Test
58 | void parse_withEmptyFleetNamePrefixAndEmptyParametersReturnsEmpty() {
59 | final EC2FleetLabelParameters parameters = new EC2FleetLabelParameters("_");
60 | assertNull(parameters.get("c"));
61 | }
62 |
63 | @Test
64 | void parse_skipParameterWithoutValue() {
65 | final EC2FleetLabelParameters parameters = new EC2FleetLabelParameters("withoutValue,b=2");
66 | assertEquals("2", parameters.get("b"));
67 | assertNull(parameters.get("withoutValue"));
68 | }
69 |
70 | @Test
71 | void parse_skipParameterWithEmptyValue() {
72 | final EC2FleetLabelParameters parameters = new EC2FleetLabelParameters("withoutValue=,b=2");
73 | assertEquals("2", parameters.get("b"));
74 | assertNull(parameters.get("withoutValue"));
75 | }
76 |
77 | }
78 |
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/EC2FleetNodeComputer.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import hudson.slaves.SlaveComputer;
4 | import org.apache.commons.lang.StringUtils;
5 | import org.kohsuke.stapler.HttpResponse;
6 |
7 | import javax.annotation.CheckForNull;
8 | import javax.annotation.Nonnull;
9 | import javax.annotation.concurrent.ThreadSafe;
10 | import java.io.IOException;
11 | import java.util.logging.Logger;
12 |
13 | /**
14 | * The {@link EC2FleetNodeComputer} represents the running state of {@link EC2FleetNode} that holds executors.
15 | * @see hudson.model.Computer
16 | */
17 | @ThreadSafe
18 | public class EC2FleetNodeComputer extends SlaveComputer {
19 | private static final Logger LOGGER = Logger.getLogger(EC2FleetNodeComputer.class.getName());
20 | private boolean isMarkedForDeletion;
21 |
22 | public EC2FleetNodeComputer(final EC2FleetNode agent) {
23 | super(agent);
24 | this.isMarkedForDeletion = false;
25 | }
26 |
27 | public boolean isMarkedForDeletion() {
28 | return isMarkedForDeletion;
29 | }
30 |
31 | @Override
32 | public EC2FleetNode getNode() {
33 | return (EC2FleetNode) super.getNode();
34 | }
35 |
36 | @CheckForNull
37 | public String getInstanceId() {
38 | EC2FleetNode node = getNode();
39 | return node == null ? null : node.getInstanceId();
40 | }
41 |
42 | public AbstractEC2FleetCloud getCloud() {
43 | final EC2FleetNode node = getNode();
44 | return node == null ? null : node.getCloud();
45 | }
46 |
47 | /**
48 | * Return label which will represent executor in "Build Executor Status"
49 | * section of Jenkins UI.
50 | *
51 | * @return Node's display name
52 | */
53 | @Nonnull
54 | @Override
55 | public String getDisplayName() {
56 | final EC2FleetNode node = getNode();
57 | if(node != null) {
58 | final int usesRemaining = node.getUsesRemaining();
59 | if(usesRemaining >= 0) {
60 | return String.format("%s Builds left: %d ", node.getDisplayName(), usesRemaining);
61 | }
62 | return node.getDisplayName();
63 | }
64 | return "unknown fleet" + " " + getName();
65 | }
66 |
67 | /**
68 | * When the agent is deleted, schedule EC2 instance for termination
69 | *
70 | * @return HttpResponse
71 | */
72 | @Override
73 | public HttpResponse doDoDelete() throws IOException {
74 | checkPermission(DELETE);
75 | final EC2FleetNode node = getNode();
76 | if (node != null) {
77 | final String instanceId = node.getInstanceId();
78 | final AbstractEC2FleetCloud cloud = node.getCloud();
79 | if (cloud != null && StringUtils.isNotBlank(instanceId)) {
80 | cloud.scheduleToTerminate(instanceId, false, EC2AgentTerminationReason.AGENT_DELETED);
81 | // Persist a flag here as the cloud objects can be re-created on user-initiated changes, hence, losing track of instance ids scheduled to terminate.
82 | this.isMarkedForDeletion = true;
83 | }
84 | }
85 | return super.doDoDelete();
86 | }
87 | }
88 |
--------------------------------------------------------------------------------
/src/test/java/com/amazon/jenkins/ec2fleet/EC2FleetLabelCloudIntegrationTest.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import hudson.model.FreeStyleProject;
4 | import hudson.model.labels.LabelAtom;
5 | import hudson.model.queue.QueueTaskFuture;
6 | import org.junit.jupiter.api.BeforeAll;
7 | import org.junit.jupiter.api.Test;
8 | import org.mockito.Mockito;
9 | import software.amazon.awssdk.services.cloudformation.CloudFormationClient;
10 | import software.amazon.awssdk.services.cloudformation.model.DeleteStackRequest;
11 | import software.amazon.awssdk.services.ec2.model.InstanceStateName;
12 |
13 | import java.util.Collections;
14 | import java.util.List;
15 | import java.util.concurrent.TimeUnit;
16 |
17 | import static org.junit.jupiter.api.Assertions.assertEquals;
18 | import static org.mockito.ArgumentMatchers.any;
19 |
20 | class EC2FleetLabelCloudIntegrationTest extends IntegrationTest {
21 |
22 | @BeforeAll
23 | static void beforeClass() {
24 | setJenkinsTestTimoutTo720();
25 | }
26 |
27 | @Test
28 | void should_create_stack_and_provision_node_for_task_execution() throws Exception {
29 | mockEc2FleetApiToEc2SpotFleet(InstanceStateName.RUNNING);
30 | mockCloudFormationApi();
31 |
32 | EC2FleetLabelCloud cloud = new EC2FleetLabelCloud("FleetLabel", "credId", "region",
33 | null, null, new LocalComputerConnector(j), false, false,
34 | 0, 0, 0, 1, false,
35 | false, 0, 0,
36 | 2, false, "test1");
37 | j.jenkins.clouds.add(cloud);
38 |
39 | // set max size to > 0 otherwise nothing to provision
40 | final String labelString = "FleetLabel_maxSize=1";
41 | final List rs = enqueTask(1, labelString, JOB_SLEEP_TIME);
42 |
43 | assertEquals(0, j.jenkins.getNodes().size());
44 |
45 | tryUntil(() -> {
46 | triggerSuggestReviewNow(labelString);
47 | assertTasksDone(rs);
48 | }, TimeUnit.MINUTES.toMillis(4));
49 |
50 | cancelTasks(rs);
51 | }
52 |
53 | @Test
54 | void should_delete_resources_if_label_unused() throws Exception {
55 | mockEc2FleetApiToEc2SpotFleet(InstanceStateName.RUNNING);
56 | final CloudFormationClient amazonCloudFormation = mockCloudFormationApi();
57 |
58 | EC2FleetLabelCloud cloud = new EC2FleetLabelCloud("FleetLabel", "credId", "region",
59 | null, null, new LocalComputerConnector(j), false, false,
60 | 0, 0, 0, 1, false,
61 | false, 0, 0,
62 | 2, false, "test1");
63 | j.jenkins.clouds.add(cloud);
64 |
65 | // set max size to > 0 otherwise nothing to provision
66 | final String labelString = "FleetLabel_maxSize=1";
67 | final List rs = enqueTask(1, labelString, JOB_SLEEP_TIME);
68 |
69 | // wait until tasks will be completed
70 | tryUntil(() -> {
71 | triggerSuggestReviewNow(labelString);
72 | assertTasksDone(rs);
73 | }, TimeUnit.MINUTES.toMillis(4));
74 |
75 | // remove label from task (unused)
76 | FreeStyleProject freeStyleProject = (FreeStyleProject) j.jenkins.getAllItems().get(0);
77 | freeStyleProject.setAssignedLabel(new LabelAtom("nothing"));
78 |
79 | // wait until stack will be deleted and nodes will be removed as well
80 | tryUntil(() -> {
81 | assertEquals(Collections.emptyList(), j.jenkins.getNodes());
82 | Mockito.verify(amazonCloudFormation).deleteStack(any(DeleteStackRequest.class));
83 | }, TimeUnit.MINUTES.toMillis(2));
84 |
85 | cancelTasks(rs);
86 | }
87 |
88 | }
89 |
--------------------------------------------------------------------------------
/docs/LABEL-BASED-CONFIGURATION.md:
--------------------------------------------------------------------------------
1 | [Back to README](../README.md)
2 |
3 | # Label Based Configuration
4 |
5 | * [Overview](#overview)
6 | * [How it works](#how-it-works)
7 | * [Supported Parameters](#supported-parameters)
8 | * [Configuration](#configuration)
9 |
10 | # Overview
11 |
12 | Feature in *beta* mode. Please report all problem [here](https://github.com/jenkinsci/ec2-fleet-plugin/issues/new)
13 |
14 | This feature auto manages EC2 Spot Fleet or ASG based Fleets for Jenkins based on
15 | label attached to Jenkins Jobs.
16 |
17 | With this feature user of EC2 Fleet Plugin doesn't need to have pre-created AWS resources
18 | to start configuration and run Jobs. Plugin required just AWS Credentials
19 | with permissions to be able create resources.
20 |
21 | # How It Works
22 |
23 | - Plugin detects all labeled Jobs where Label starts from Name configured in plugin configuration ```Cloud Name```
24 | - Plugin parses Label to get Fleet configuration
25 | - Plugin creates dedicated fleet for each unique Label
26 | - Plugin uses [CloudFormation Stacks](https://aws.amazon.com/cloudformation/) to provision Fleet and all required resources
27 | - When Label is not used by any Job Plugin deletes Stack and release resources
28 |
29 | Label format
30 | ```
31 | _parameter1=value1,parameter2=value2
32 | ```
33 |
34 | # Supported Parameters
35 |
36 | *Note* Parameter name is case insensitive
37 |
38 | | Parameter | Value Example | Value |
39 | | --- | ---| ---- |
40 | | imageId | ```ami-0080e4c5bc078760e``` | *Required* AMI ID https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html |
41 | | max | ```12``` | Fleet Max Size, positive value or zero. If not specified plugin configuration Max will be used |
42 | | min | ```1``` | Fleet Min Size, positive value or zero. If not specified plugin configuration Min will be used |
43 | | instanceType | ```c4.large``` | EC2 Instance Type https://aws.amazon.com/ec2/instance-types/. If not specified ```m4.large``` will be used |
44 | | spotPrice | ```0.4``` | Max Spot Price, if not specified EC2 Spot Fleet API will use default price. |
45 |
46 | ### Examples
47 |
48 | Minimum configuration just Image ID
49 | ```
50 | _imageId=ami-0080e4c5bc078760e
51 | ```
52 |
53 | # Configuration
54 |
55 | 1. Create AWS User. _Alternatively, you can use an [AWS EC2 instance role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)_
56 | 1. Add Inline User Permissions
57 | ```json
58 | {
59 | "Version": "2012-10-17",
60 | "Statement": [{
61 | "Effect": "Allow",
62 | "Action": [
63 | "cloudformation:*",
64 | "ec2:*",
65 | "autoscaling:*",
66 | "iam:ListRoles",
67 | "iam:PassRole",
68 | "iam:ListInstanceProfiles",
69 | "iam:CreateRole",
70 | "iam:AttachRolePolicy",
71 | "iam:GetRole"
72 | ],
73 | "Resource": "*"
74 | }]
75 | }
76 | ```
77 | 1. Goto ```Manage Jenkins > Configure Jenkins```
78 | 1. Add Cloud ```Amazon EC2 Fleet label based```
79 | 1. Specify ```AWS Credentials```
80 | 1. Specify ```SSH Credentials```
81 | - Jenkins need to be able to connect to EC2 Instances to run Jobs
82 | 1. Set ```Region```
83 | 1. Provide base configuration
84 | - Note ```Cloud Name```
85 | 1. Goto to Jenkins Job which you want to run on this Fleet
86 | 1. Goto Job ```Configuration```
87 | 1. Enable ```Restrict where this project can be run```
88 | 1. Set Label value to ```_parameterName=paremeterValue,p2=v2```
89 | 1. Click ```Save```
90 |
91 | In some short time plugin will detect Job and will create required resources to be able
92 | run it in future.
93 |
94 | That's all, you can repeat this for other Jobs.
95 |
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/EC2FleetNode.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import hudson.Extension;
4 | import hudson.model.Computer;
5 | import hudson.model.Descriptor;
6 | import hudson.model.Failure;
7 | import hudson.model.Node;
8 | import hudson.model.Slave;
9 | import hudson.slaves.ComputerLauncher;
10 | import hudson.slaves.EphemeralNode;
11 | import hudson.slaves.NodeProperty;
12 | import hudson.slaves.RetentionStrategy;
13 | import jenkins.model.Jenkins;
14 |
15 | import java.io.IOException;
16 | import java.util.List;
17 | import java.util.logging.Logger;
18 |
19 | /**
20 | * The {@link EC2FleetNode} represents an agent running on an EC2 instance, responsible for creating {@link EC2FleetNodeComputer}.
21 | */
22 | public class EC2FleetNode extends Slave implements EphemeralNode {
23 | private static final Logger LOGGER = Logger.getLogger(EC2FleetNode.class.getName());
24 |
25 | private String cloudName;
26 | private String instanceId;
27 | private final int maxTotalUses;
28 | private int usesRemaining;
29 |
30 | public EC2FleetNode(final String instanceId, final String nodeDescription, final String remoteFS, final int numExecutors, final Mode mode, final String label,
31 | final List> nodeProperties, final String cloudName, ComputerLauncher launcher, final int maxTotalUses) throws IOException, Descriptor.FormException {
32 | //noinspection deprecation
33 | super(instanceId, nodeDescription, remoteFS, numExecutors, mode, label,
34 | launcher, RetentionStrategy.NOOP, nodeProperties);
35 |
36 | this.cloudName = cloudName;
37 | this.instanceId = instanceId;
38 | this.maxTotalUses = maxTotalUses;
39 | this.usesRemaining = maxTotalUses;
40 | }
41 |
42 | public String getCloudName() {
43 | return cloudName;
44 | }
45 |
46 | public String getInstanceId() {
47 | return instanceId;
48 | }
49 |
50 | public void setInstanceId(String instanceId) {
51 | this.instanceId = instanceId;
52 | }
53 |
54 | public int getMaxTotalUses() {
55 | return this.maxTotalUses;
56 | }
57 |
58 | public int getUsesRemaining() {
59 | return usesRemaining;
60 | }
61 |
62 | public void decrementUsesRemaining() {
63 | this.usesRemaining--;
64 | }
65 |
66 | @Override
67 | public Node asNode() {
68 | return this;
69 | }
70 |
71 | @Override
72 | public String getDisplayName() {
73 | final String name = String.format("%s %s", cloudName, instanceId);
74 | try {
75 | Jenkins.checkGoodName(name);
76 | return name;
77 | } catch (Failure e) {
78 | return instanceId;
79 | }
80 | }
81 |
82 | @Override
83 | public Computer createComputer() {
84 | return new EC2FleetNodeComputer(this);
85 | }
86 |
87 | public AbstractEC2FleetCloud getCloud() {
88 | return (AbstractEC2FleetCloud) Jenkins.get().getCloud(cloudName);
89 | }
90 |
91 | public DescriptorImpl getDescriptor() {
92 | return (DescriptorImpl) super.getDescriptor();
93 | }
94 |
95 | @Extension
96 | public static final class DescriptorImpl extends SlaveDescriptor {
97 |
98 | public DescriptorImpl() {
99 | super();
100 | }
101 |
102 | public String getDisplayName() {
103 | return "Fleet Slave";
104 | }
105 |
106 | /**
107 | * We only create this kind of nodes programmatically.
108 | */
109 | @Override
110 | public boolean isInstantiable() {
111 | return false;
112 | }
113 | }
114 | }
115 |
--------------------------------------------------------------------------------
/CONTRIBUTING.md:
--------------------------------------------------------------------------------
1 | # Contributing Guidelines
2 |
3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional
4 | documentation, we greatly value feedback and contributions from our community.
5 |
6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
7 | information to effectively respond to your bug report or contribution.
8 |
9 |
10 | ## Reporting Bugs/Feature Requests
11 |
12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features.
13 |
14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already
15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
16 |
17 | * A reproducible test case or series of steps
18 | * The version of our code being used
19 | * Logs associated with the plugin
20 | * Any modifications you've made relevant to the bug
21 | * Anything unusual about your environment or deployment
22 |
23 |
24 | ## Contributing via Pull Requests
25 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
26 |
27 | 1. You are working against the latest source on the *master* branch.
28 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
29 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
30 |
31 | To send us a pull request, please:
32 |
33 | 1. Fork the repository.
34 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
35 | 3. Ensure local tests pass. Integration Tests will also be executed along with unit tests. Execute tests by running following command: `mvn test`
36 | 4. Commit to your fork using clear commit messages.
37 | 5. Send us a pull request, answering any default questions in the pull request interface.
38 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
39 |
40 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
41 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
42 |
43 | ## Run Jenkins Plugin Locally
44 | Prerequisites:
45 | To develop a plugin, you need Maven 3 and JDK 6.0 or later
46 |
47 | We recommend you to test your changes locally before making your contributions. You can test the behavior of the plugin by running it locally after making changes by running following comand:
48 | ```
49 | # Start Jenkins on default local port: 8080
50 | $ mvn hpi:run
51 |
52 | # If you need to launch the Jenkins on a different port than 8080, set the port through the system property jetty.port.
53 | $ mvn hpi:run -Djetty.port=8090
54 | ```
55 |
56 | ## Finding contributions to work on
57 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.
58 |
59 |
60 | ## Code of Conduct
61 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
62 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
63 | opensource-codeofconduct@amazon.com with any additional questions or comments.
64 |
65 |
66 | ## Security issue notifications
67 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
68 |
69 | ## Licensing
70 |
71 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/aws/RegionHelper.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet.aws;
2 |
3 | import com.amazon.jenkins.ec2fleet.Registry;
4 | import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
5 | import hudson.util.ListBoxModel;
6 | import software.amazon.awssdk.services.ec2.Ec2Client;
7 | import software.amazon.awssdk.services.ec2.model.DescribeRegionsResponse;
8 | import software.amazon.awssdk.services.ec2.model.Region;
9 |
10 | import java.util.TreeMap;
11 | import java.util.stream.Collectors;
12 |
13 | public class RegionHelper {
14 |
15 | /**
16 | * Fill Regions
17 | *
18 | * Get region codes (e.g. us-east-1) from EC2 API and AWS SDK.
19 | * DescribeRegions API does not have region descriptions (such as us-east-1 - US East (N. Virginia))
20 | * We fetch descriptions from our RegionInfo enum to avoid unnecessarily upgrading
21 | * AWS Java SDK for newer regions and fallback to AWS Java SDK enum.
22 | *
23 | * @param awsCredentialsId aws credentials id
24 | * @return ListBoxModel with label and values
25 | */
26 | @SuppressFBWarnings(
27 | value = {"DE_MIGHT_IGNORE", "WMI_WRONG_MAP_ITERATOR"},
28 | justification = "Ignore API exceptions and key iterator is really intended")
29 | public static ListBoxModel getRegionsListBoxModel(final String awsCredentialsId) {
30 | // to keep user consistent order tree map, default value to regionCode (eg. us-east-1)
31 | final TreeMap regionDisplayNames = new TreeMap<>();
32 | try {
33 | final Ec2Client client = Registry.getEc2Api().connect(awsCredentialsId, null, null);
34 | final DescribeRegionsResponse regions = client.describeRegions();
35 | regionDisplayNames.putAll(regions.regions().stream()
36 | .collect(Collectors.toMap(Region::regionName, Region::regionName)));
37 | } catch (final Exception ex) {
38 | // ignore exception it could be case that credentials are not belong to default region
39 | // which we are using to describe regions
40 | }
41 | // Add SDK regions as user can have latest SDK
42 | regionDisplayNames.putAll(software.amazon.awssdk.regions.Region.regions().stream()
43 | .collect(Collectors.toMap(software.amazon.awssdk.regions.Region::id, software.amazon.awssdk.regions.Region::id)));
44 | // Add regions from enum as user may have older SDK
45 | regionDisplayNames.putAll(RegionInfo.getRegionNames().stream()
46 | .collect(Collectors.toMap(r -> r, r -> r)));
47 |
48 | final ListBoxModel model = new ListBoxModel();
49 | for (final String regionName : regionDisplayNames.keySet()) {
50 | String regionDescription;
51 | try {
52 | final RegionInfo region = RegionInfo.fromName(regionName);
53 | if (region != null) {
54 | regionDescription = region.getDescription();
55 | } else {
56 | // Fallback to SDK when region description not found in RegionInfo
57 | software.amazon.awssdk.regions.Region sdkRegion = software.amazon.awssdk.regions.Region.of(regionName);
58 | if (sdkRegion != null && sdkRegion.metadata() != null && sdkRegion.metadata().description() != null) {
59 | regionDescription = sdkRegion.metadata().description();
60 | } else {
61 | // If metadata or description is missing, use region code
62 | regionDescription = null;
63 | }
64 | }
65 | final String regionDisplayName = regionDescription != null ? String.format("%s %s", regionName, regionDescription) : regionName;
66 |
67 | // Update map only when description exists else leave default to region code eg. us-east-1
68 | regionDisplayNames.put(regionName, regionDisplayName);
69 | } catch (final IllegalArgumentException ex) {
70 | // Description missing in both enum and SDK, ignore and leave default
71 | }
72 | model.add(new ListBoxModel.Option(regionDisplayNames.get(regionName), regionName));
73 | }
74 | return model;
75 | }
76 | }
77 |
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/aws/RegionInfo.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet.aws;
2 |
3 | import java.util.ArrayList;
4 | import java.util.List;
5 |
6 | /**
7 | * Copied from SDK to avoid upgrading SDK for newer regions
8 | */
9 | public enum RegionInfo {
10 | GovCloud("us-gov-west-1", "AWS GovCloud (US-West)"),
11 | US_GOV_EAST_1("us-gov-east-1", "AWS GovCloud (US-East)"),
12 | US_EAST_1("us-east-1", "US East (N. Virginia)"),
13 | US_EAST_2("us-east-2", "US East (Ohio)"),
14 | US_WEST_1("us-west-1", "US West (N. California)"),
15 | US_WEST_2("us-west-2", "US West (Oregon)"),
16 | EU_WEST_1("eu-west-1", "Europe (Ireland)"),
17 | EU_WEST_2("eu-west-2", "Europe (London)"),
18 | EU_WEST_3("eu-west-3", "Europe (Paris)"),
19 | EU_CENTRAL_1("eu-central-1", "Europe (Frankfurt)"),
20 | EU_CENTRAL_2("eu-central-2", "Europe (Zurich)"),
21 | EU_NORTH_1("eu-north-1", "Europe (Stockholm)"),
22 | EU_SOUTH_1("eu-south-1", "Europe (Milan)"),
23 | EU_SOUTH_2("eu-south-2", "Europe (Spain)"),
24 | AP_EAST_1("ap-east-1", "Asia Pacific (Hong Kong)"),
25 | AP_SOUTH_1("ap-south-1", "Asia Pacific (Mumbai)"),
26 | AP_SOUTH_2("ap-south-2", "Asia Pacific (Hyderabad)"),
27 | AP_SOUTHEAST_1("ap-southeast-1", "Asia Pacific (Singapore)"),
28 | AP_SOUTHEAST_2("ap-southeast-2", "Asia Pacific (Sydney)"),
29 | AP_SOUTHEAST_3("ap-southeast-3", "Asia Pacific (Jakarta)"),
30 | AP_SOUTHEAST_4("ap-southeast-4", "Asia Pacific (Melbourne)"),
31 | AP_NORTHEAST_1("ap-northeast-1", "Asia Pacific (Tokyo)"),
32 | AP_NORTHEAST_2("ap-northeast-2", "Asia Pacific (Seoul)"),
33 | AP_NORTHEAST_3("ap-northeast-3", "Asia Pacific (Osaka)"),
34 |
35 | SA_EAST_1("sa-east-1", "South America (Sao Paulo)"),
36 | CN_NORTH_1("cn-north-1", "China (Beijing)"),
37 | CN_NORTHWEST_1("cn-northwest-1", "China (Ningxia)"),
38 | CA_CENTRAL_1("ca-central-1", "Canada (Central)"),
39 | CA_WEST_1("ca-west-1", "Canada West (Calgary)"),
40 | ME_CENTRAL_1("me-central-1", "Middle East (UAE)"),
41 | ME_SOUTH_1("me-south-1", "Middle East (Bahrain)"),
42 | AF_SOUTH_1("af-south-1", "Africa (Cape Town)"),
43 | US_ISO_EAST_1("us-iso-east-1", "US ISO East"),
44 | US_ISOB_EAST_1("us-isob-east-1", "US ISOB East (Ohio)"),
45 | US_ISO_WEST_1("us-iso-west-1", "US ISO WEST"),
46 | IL_CENTRAL_1("il-central-1", "Israel (Tel Aviv)"),
47 | AWS_CN_GLOBAL("aws-cn-global", "aws-cn global region"),
48 | US_ISOF_SOUTH_1("us-isof-south-1", "US ISOF SOUTH"),
49 | AP_EAST_2("ap-east-2", "Asia Pacific (Taipei)"),
50 | AP_SOUTHEAST_5("ap-southeast-5", "Asia Pacific (Malaysia)"),
51 | AP_SOUTHEAST_7("ap-southeast-7", "Asia Pacific (Thailand)"),
52 | AWS_ISO_E_GLOBAL("aws-iso-e-global", "aws-iso-e global region"),
53 | MX_CENTRAL_1("mx-central-1", "Mexico (Central)"),
54 | EUSC_DE_EAST_1("eusc-de-east-1", "EU (Germany)"),
55 | EU_ISOE_WEST_1("eu-isoe-west-1", "EU ISOE West"),
56 | AWS_GLOBAL("aws-global", "aws global region"),
57 | AWS_ISO_GLOBAL("aws-iso-global", "aws-iso global region"),
58 | AWS_ISO_B_GLOBAL("aws-iso-b-global", "aws-iso-b global region"),
59 | AWS_ISO_F_GLOBAL("aws-iso-f-global", "aws-iso-f global region"),
60 | AWS_US_GOV_GLOBAL("aws-us-gov-global", "aws-us-gov global region"),
61 | US_ISOF_EAST_1("us-isof-east-1", "US ISOF EAST"),
62 | AP_SOUTHEAST_6("ap-southeast-6", "Asia Pacific (New Zealand)");
63 |
64 | private final String name;
65 | private final String description;
66 |
67 | private RegionInfo(String name, String description) {
68 | this.name = name;
69 | this.description = description;
70 | }
71 |
72 | public String getName() {
73 | return this.name;
74 | }
75 |
76 | public String getDescription() {
77 | return this.description;
78 | }
79 |
80 | public static RegionInfo fromName(String regionName) {
81 | for (final RegionInfo region : values()) {
82 | if (region.getName().equalsIgnoreCase(regionName)) {
83 | return region;
84 | }
85 | }
86 | return null;
87 | }
88 |
89 | public static List getRegionNames() {
90 | final List regionNames = new ArrayList<>();
91 | for(final RegionInfo regionInfo : values()) {
92 | regionNames.add(regionInfo.getName());
93 | }
94 | return regionNames;
95 | }
96 | }
97 |
--------------------------------------------------------------------------------
/docs/SETUP-WINDOWS-AGENT.md:
--------------------------------------------------------------------------------
1 | # Windows Agent with EC2 Fleet Plugin
2 |
3 | This guide describes how to configure Windows EC2 Instance to be good for run
4 | as Agent for EC2 Fleet Jenkins Plugin. At the end of this guide you
5 | will get AWS EC2 AMI (Image) which could be used for Auto Scaling Group
6 | or EC2 Spot Fleet to run Windows agents.
7 |
8 | **Big thanks to @Michenux for help to find all details**
9 |
10 | **Note** Before this, please consider to use Windows OpenSSH
11 | https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/CONFIGURE.md#launch-windows-slaves-using-microsoft-openssh
12 |
13 | **Note** This guide uses Windows DCOM technology (not open ssh) it doesn't work over NAT,
14 | so Jenkins Master EC2 Instance should be placed in same VPC as Agents managed by EC2 Fleet Plugin.
15 |
16 | ## Run EC2 Instance with Windows
17 |
18 | 1. Note Windows Password for this guide
19 | 1. Login to Windows
20 |
21 | ## Create Jenkins User
22 |
23 | 1. Goto ```Local Users and Groups```
24 | 1. Click ```Users```
25 | 1. Create New with name ```jenkins```
26 | - Set password and note it
27 | - Set ```Password never expires```
28 | - Set ```User cannot change password```
29 | - Unset ```User must change password at next logon```
30 | 1. Goto user properties, find ```Member Of``` add ```Administrators``` group
31 |
32 | ## Login to Windows as jenkins user
33 |
34 | ### Configure Windows Registry
35 |
36 | 1. Run ```regedit```
37 |
38 | 1. Set ```HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled``` to ```1```
39 |
40 | 1. Goto ```HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System```
41 | 1. Create/Modify ```DWORD-32``` with name ```LocalAccountTokenFilterPolicy``` value ```1```
42 |
43 | 1. Goto ```HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa```
44 | 1. Create/Modify ```DWORD-32``` with name ```LMCompatibilityLevel``` value ```2```
45 | - send NTLM authentication only
46 |
47 | 1. Find key ```76A64158-CB41-11D1-8B02-00600806D9B6```
48 | - it’s in ```HKEY_CLASSES_ROOT\CLSID```
49 | 1. Right click and select ```Permissions```
50 | 1. Change owner to ```Administrators``` select apply to children
51 | 1. Add ```Full Control``` to ```Administrators``` make sure to apply for children as well
52 | 1. Change owner back to ```NT Service\TrustedInstaller``` select apply to children
53 |
54 | 1. Run service ```Remote Registry```
55 | 1. Restart Windows
56 |
57 | ### Configure smb
58 |
59 | 1. Run as ```PowerShell``` as Administrator
60 | 1. Run ```Enable-WindowsOptionalFeature -Online -FeatureName smb1protocol```
61 | 1. Run ```Set-SmbServerConfiguration -EnableSMB1Protocol $true```
62 |
63 | ### Configure Firewall
64 |
65 | 1. Search for ```Windows Defender Firewall```
66 | 1. Click ```Advanced settings```
67 | 1. Goto ```Inbound Rules```
68 | 1. Add ```Remote Assistance TCP 135```
69 | 1. Add ```File and Printer Sharing (NB-Name-In) UDP 137```
70 | 1. Add ```File and Printer Sharing (NB-Datagram-In) UDP 138```
71 | 1. Add ```File and Printer Sharing (NB-Session-In) TCP 139```
72 | 1. Add ```File and Printer Sharing (SMB-In) TCP 445```
73 | 1. Add ```jenkins-master 40000-60000 TCP 40000-60000```
74 | 1. Add ```Administrator at Distance COM+ (DCOM) TCP C:\WINDOWS\System32\dllhost.exe```
75 | 1. For all created goto ```Properties -> Advanced``` and set ```Allow edge traversal```
76 |
77 | ## Install Java
78 |
79 | 1. Open ```PowerShell```
80 | 1. Install [Scoop](https://scoop.sh/) ```Invoke-Expression (New-Object System.Net.WebClient).DownloadString('https://get.scoop.sh')```
81 | ```scoop install git-with-openssh```
82 | 1. ```scoop bucket add java```
83 | 1. ```scoop install ojdkbuild8-full```
84 |
85 | ### Configure System Path for Java
86 |
87 | 1. Goto ```Control Panel\System and Security\System```
88 | 1. Goto ```Advanced System Settings```
89 | 1. Goto ```Environment Variables...```
90 | 1. Add Java Path (```C:\Users\jenkins\scoop\apps\ojdkbuild8-full\current\bin``` installed before by scoop) to System ```PATH```
91 |
92 | ## Create EC2 AMI
93 |
94 | 1. Goto to AWS Console and create image of preconfigured instance
95 |
96 | ## Before using this AMI for Jenkins Agent
97 |
98 | - Make sure you required traffic could go to Windows from Jenkins. You can find
99 | required ports above in ```Configure Firewall``` section
100 |
101 | ## Troubleshooting
102 |
103 | - https://github.com/jenkinsci/windows-slaves-plugin/blob/35b7f1d77b612af2c45b558b03538d0fb53fc05b/docs/troubleshooting.adoc
--------------------------------------------------------------------------------
/src/test/java/com/amazon/jenkins/ec2fleet/ProvisionPerformanceTest.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import hudson.model.FreeStyleBuild;
4 | import hudson.model.Result;
5 | import hudson.model.queue.QueueTaskFuture;
6 | import hudson.slaves.ComputerConnector;
7 | import org.junit.jupiter.api.BeforeAll;
8 | import org.junit.jupiter.api.Disabled;
9 | import org.junit.jupiter.api.Test;
10 | import software.amazon.awssdk.services.ec2.model.InstanceStateName;
11 |
12 | import java.io.IOException;
13 | import java.util.ArrayList;
14 | import java.util.List;
15 | import java.util.concurrent.ExecutionException;
16 | import java.util.concurrent.TimeUnit;
17 |
18 | import static org.hamcrest.MatcherAssert.assertThat;
19 | import static org.hamcrest.Matchers.lessThanOrEqualTo;
20 | import static org.junit.jupiter.api.Assertions.assertEquals;
21 | import static org.junit.jupiter.api.Assertions.assertNotNull;
22 |
23 | @Disabled
24 | class ProvisionPerformanceTest extends IntegrationTest {
25 |
26 | private final EC2FleetCloud.ExecutorScaler noScaling = new EC2FleetCloud.NoScaler();
27 |
28 | @BeforeAll
29 | static void beforeClass() {
30 | System.setProperty("jenkins.test.timeout", "720");
31 | }
32 |
33 | @Test
34 | void spikeLoadWorkers10Tasks30() throws Exception {
35 | test(10, 30);
36 | }
37 |
38 | @Test
39 | void spikeLoadWorkers20Tasks60() throws Exception {
40 | test(20, 60);
41 | }
42 |
43 | private void test(int workers, int maxTasks) throws IOException {
44 | mockEc2FleetApiToEc2SpotFleetWithDelay(InstanceStateName.RUNNING, 500);
45 |
46 | final ComputerConnector computerConnector = new LocalComputerConnector(j);
47 | final EC2FleetCloudWithMeter cloud = new EC2FleetCloudWithMeter(null, "credId", null, "region",
48 | null, "fId", "momo", null, computerConnector, false, false,
49 | 1, 0, workers, 0, 1, true, false,
50 | false, 0, 0, 2, false, noScaling);
51 | j.jenkins.clouds.add(cloud);
52 |
53 | // updated plugin requires some init time to get first update
54 | // so wait this event to be really correct with perf comparison as old version is not require init time
55 | tryUntil(() -> assertNotNull(cloud.getStats()));
56 |
57 | System.out.println("start test");
58 | final long start = System.currentTimeMillis();
59 |
60 | final List> tasks = new ArrayList<>();
61 |
62 | final int taskBatch = 5;
63 |
64 | while (tasks.size() < maxTasks) {
65 | tasks.addAll((List) enqueTask(taskBatch));
66 | triggerSuggestReviewNow("momo");
67 | System.out.println(taskBatch + " added into queue, " + (maxTasks - tasks.size()) + " remain");
68 | }
69 |
70 | for (final QueueTaskFuture task : tasks) {
71 | try {
72 | assertEquals(Result.SUCCESS, task.get().getResult());
73 | } catch (InterruptedException | ExecutionException e) {
74 | throw new RuntimeException(e);
75 | }
76 | }
77 |
78 | System.out.println("downscale");
79 | final long finish = System.currentTimeMillis();
80 |
81 | // wait until downscale happens
82 | tryUntil(() -> {
83 | // defect in termination logic, that why 1
84 | assertThat(j.jenkins.getLabel("momo").getNodes().size(), lessThanOrEqualTo(1));
85 | }, TimeUnit.MINUTES.toMillis(3));
86 |
87 | final long upTime = TimeUnit.MILLISECONDS.toSeconds(finish - start);
88 | final long downTime = TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis() - finish);
89 | final long totalTime = upTime + downTime;
90 | final long ideaUpTime = (maxTasks / workers) * JOB_SLEEP_TIME;
91 | final int idealDownTime = 60;
92 | final long ideaTime = ideaUpTime + idealDownTime;
93 |
94 | System.out.println(maxTasks + " up in " + upTime + " sec, ideal time is " + ideaUpTime + " sec, overhead is " + (upTime - ideaUpTime) + " sec");
95 | System.out.println(maxTasks + " down in " + downTime + " sec, ideal time is " + idealDownTime + " sec, overhead is " + (downTime - idealDownTime) + " sec");
96 | System.out.println(maxTasks + " completed in " + totalTime + " sec, ideal time is " + ideaTime + " sec, overhead is " + (totalTime - ideaTime) + " sec");
97 | System.out.println(cloud.provisionMeter);
98 | System.out.println(cloud.removeMeter);
99 | System.out.println(cloud.updateMeter);
100 | }
101 |
102 | }
103 |
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/EC2FleetOnlineChecker.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import hudson.model.Computer;
4 | import hudson.model.Node;
5 | import hudson.util.DaemonThreadFactory;
6 |
7 | import javax.annotation.concurrent.ThreadSafe;
8 | import java.util.concurrent.CompletableFuture;
9 | import java.util.concurrent.Executors;
10 | import java.util.concurrent.ScheduledExecutorService;
11 | import java.util.concurrent.TimeUnit;
12 | import java.util.logging.Level;
13 | import java.util.logging.Logger;
14 |
15 | /**
16 | * Keep {@link hudson.slaves.NodeProvisioner.PlannedNode#future} not resolved until node will not be online
17 | * or timeout reached.
18 | *
19 | * Default Jenkins node capacity planner {@link hudson.slaves.NodeProvisioner.Strategy} count planned nodes
20 | * as available capacity, but exclude offline computers {@link Computer#isOnline()} from available capacity.
21 | * Because EC2 instance requires some time when it was added into fleet to start up, next situation happens:
22 | * plugin add described capacity as node into Jenkins pool, but Jenkins keeps it as offline as no way to connect,
23 | * during time when node is offline, Jenkins will try to request more nodes from plugin as offline nodes
24 | * excluded from capacity.
25 | *
26 | * This class fix this situation and keep planned node until instance is really online, so Jenkins planner
27 | * count planned node as available capacity and doesn't request more.
28 | *
29 | * Before each wait it will try to {@link Computer#connect(boolean)}, because by default Jenkins is trying to
30 | * make a few short interval reconnection initially (when EC2 instance still is not ready) after that
31 | * with big interval, experiment shows a few minutes and more.
32 | *
24 | * Run example:
25 | * https://docs.google.com/spreadsheets/d/e/2PACX-1vSuPWeDJD8xAbvHpyJPigAIMYJyL0YvljjAatutNqaFqUQofTx2PxY-sfqZgfsWqRxMGl2elJErbH5n/pubchart?oid=983520837&format=interactive
26 | */
27 | @Disabled
28 | class NoDelayProvisionStrategyPerformanceTest extends IntegrationTest {
29 | private final EC2FleetCloud.ExecutorScaler noScaling = new EC2FleetCloud.NoScaler();
30 |
31 | @BeforeAll
32 | static void beforeClass() {
33 | turnOffJenkinsTestTimout();
34 | // set default MARGIN for Jenkins
35 | System.setProperty(NodeProvisioner.class.getName() + ".MARGIN", Integer.toString(10));
36 | }
37 |
38 | @Test
39 | void noDelayProvisionStrategy() throws Exception {
40 | test(true);
41 | }
42 |
43 | @Test
44 | void defaultProvisionStrategy() throws Exception {
45 | test(false);
46 | }
47 |
48 | private void test(final boolean noDelay) throws IOException, InterruptedException {
49 | final int maxWorkers = 100;
50 | final int scheduleInterval = 15;
51 | final int batchSize = 9;
52 |
53 | mockEc2FleetApiToEc2SpotFleetWithDelay(InstanceStateName.RUNNING, 500);
54 |
55 | final ComputerConnector computerConnector = new LocalComputerConnector(j);
56 | final String label = "momo";
57 | final EC2FleetCloudWithHistory cloud = new EC2FleetCloudWithHistory(null, "credId", null, "region",
58 | null, "fId", label, null, computerConnector, false, false,
59 | 1, 0, maxWorkers, 0, 1, true, false,
60 | false, 0, 0,
61 | 15, noDelay, noScaling);
62 | j.jenkins.clouds.add(cloud);
63 |
64 | System.out.println("waiting cloud start");
65 | // updated plugin requires some init time to get first update
66 | // so wait this event to be really correct with perf comparison as old version is not require init time
67 | tryUntil(() -> assertNotNull(cloud.getStats()));
68 |
69 | // warm up jenkins queue, as it takes some time when jenkins run first task and start scale in/out
70 | // so let's run one task and wait it finish
71 | System.out.println("waiting warm up task execution");
72 | final List warmUpTasks = enqueTask(1);
73 | waitTasksFinish(warmUpTasks);
74 |
75 | final List> metrics = new ArrayList<>();
76 | final Thread monitor = new Thread(() -> {
77 | while (!Thread.interrupted()) {
78 | final int queueSize = j.jenkins.getQueue().countBuildableItems() // tasks to build
79 | + j.jenkins.getQueue().getPendingItems().size() // tasks to start
80 | + j.jenkins.getLabelAtom(label).getBusyExecutors(); // tasks in progress
81 | final int executors = j.jenkins.getLabelAtom(label).getTotalExecutors();
82 | final ImmutableTriple data = new ImmutableTriple<>(
83 | System.currentTimeMillis(), queueSize, executors);
84 | metrics.add(data);
85 | System.out.println(new Date(data.left) + " " + data.middle + " " + data.right);
86 |
87 | try {
88 | Thread.sleep(TimeUnit.SECONDS.toMillis(5));
89 | } catch (InterruptedException e) {
90 | throw new RuntimeException("stopped");
91 | }
92 | }
93 | });
94 | monitor.start();
95 |
96 | System.out.println("start test");
97 | int taskCount = 0;
98 | final List tasks = new ArrayList<>();
99 | for (int i = 0; i < 15; i++) {
100 | tasks.addAll(enqueTask(batchSize));
101 | taskCount += batchSize;
102 | System.out.println("schedule " + taskCount + " tasks, waiting " + scheduleInterval + " sec");
103 | Thread.sleep(TimeUnit.SECONDS.toMillis(scheduleInterval));
104 | }
105 |
106 | waitTasksFinish(tasks);
107 |
108 | monitor.interrupt();
109 | monitor.join();
110 |
111 | for (ImmutableTriple data : metrics) {
112 | System.out.println(data.middle + " " + data.right);
113 | }
114 | }
115 |
116 | private static void waitTasksFinish(List tasks) {
117 | for (final QueueTaskFuture task : tasks) {
118 | try {
119 | task.get();
120 | } catch (InterruptedException | ExecutionException e) {
121 | throw new RuntimeException(e);
122 | }
123 | }
124 | }
125 |
126 | }
127 |
--------------------------------------------------------------------------------
/src/test/java/com/amazon/jenkins/ec2fleet/EC2FleetCloudConfigurationAsCodeTest.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import com.amazon.jenkins.ec2fleet.fleet.EC2Fleet;
4 | import com.amazon.jenkins.ec2fleet.fleet.EC2Fleets;
5 | import hudson.plugins.sshslaves.SSHConnector;
6 | import hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy;
7 | import io.jenkins.plugins.casc.ConfiguratorException;
8 | import io.jenkins.plugins.casc.misc.ConfiguredWithCode;
9 | import io.jenkins.plugins.casc.misc.JenkinsConfiguredWithCodeRule;
10 | import io.jenkins.plugins.casc.misc.junit.jupiter.WithJenkinsConfiguredWithCode;
11 | import org.junit.jupiter.api.BeforeEach;
12 | import org.junit.jupiter.api.Test;
13 |
14 | import java.util.Arrays;
15 | import java.util.Collections;
16 | import java.util.HashSet;
17 |
18 | import static org.junit.jupiter.api.Assertions.*;
19 | import static org.mockito.ArgumentMatchers.anyString;
20 | import static org.mockito.ArgumentMatchers.nullable;
21 | import static org.mockito.Mockito.mock;
22 | import static org.mockito.Mockito.when;
23 |
24 | @WithJenkinsConfiguredWithCode
25 | class EC2FleetCloudConfigurationAsCodeTest {
26 |
27 | @BeforeEach
28 | void before() {
29 | final EC2Fleet ec2Fleet = mock(EC2Fleet.class);
30 | EC2Fleets.setGet(ec2Fleet);
31 | when(ec2Fleet.getState(anyString(), anyString(), nullable(String.class), anyString()))
32 | .thenReturn(new FleetStateStats("", 2, FleetStateStats.State.active(), new HashSet<>(Arrays.asList("i-1", "i-2")), Collections.emptyMap()));
33 | }
34 |
35 | @Test
36 | @ConfiguredWithCode(
37 | value = "EC2FleetCloud/name-required-configuration-as-code.yml",
38 | expected = ConfiguratorException.class,
39 | message = "name is required to configure class com.amazon.jenkins.ec2fleet.EC2FleetCloud")
40 | void configurationWithNullName_shouldFail(JenkinsConfiguredWithCodeRule jenkinsRule) {
41 | // NOP
42 | }
43 |
44 | @Test
45 | @ConfiguredWithCode("EC2FleetCloud/min-configuration-as-code.yml")
46 | void shouldCreateCloudFromMinConfiguration(JenkinsConfiguredWithCodeRule jenkinsRule) {
47 | assertEquals(1, jenkinsRule.jenkins.clouds.size());
48 | EC2FleetCloud cloud = (EC2FleetCloud) jenkinsRule.jenkins.clouds.getByName("ec2-fleet");
49 |
50 | assertEquals("ec2-fleet", cloud.name);
51 | assertNull(cloud.getRegion());
52 | assertNull(cloud.getEndpoint());
53 | assertNull(cloud.getFleet());
54 | assertNull(cloud.getFsRoot());
55 | assertFalse(cloud.isPrivateIpUsed());
56 | assertFalse(cloud.isAlwaysReconnect());
57 | assertNull(cloud.getLabelString());
58 | assertEquals(0, cloud.getIdleMinutes());
59 | assertEquals(0, cloud.getMinSize());
60 | assertEquals(0, cloud.getMaxSize());
61 | assertEquals(1, cloud.getNumExecutors());
62 | assertFalse(cloud.isAddNodeOnlyIfRunning());
63 | assertFalse(cloud.isRestrictUsage());
64 | assertEquals(EC2FleetCloud.NoScaler.class, cloud.getExecutorScaler().getClass());
65 | assertEquals(180, cloud.getInitOnlineTimeoutSec());
66 | assertEquals(15, cloud.getInitOnlineCheckIntervalSec());
67 | assertEquals(10, cloud.getCloudStatusIntervalSec());
68 | assertFalse(cloud.isDisableTaskResubmit());
69 | assertFalse(cloud.isNoDelayProvision());
70 | }
71 |
72 | @Test
73 | @ConfiguredWithCode("EC2FleetCloud/max-configuration-as-code.yml")
74 | void shouldCreateCloudFromMaxConfiguration(JenkinsConfiguredWithCodeRule jenkinsRule) {
75 | assertEquals(1, jenkinsRule.jenkins.clouds.size());
76 | EC2FleetCloud cloud = (EC2FleetCloud) jenkinsRule.jenkins.clouds.getByName("ec2-fleet");
77 |
78 | assertEquals("ec2-fleet", cloud.name);
79 | assertEquals("us-east-2", cloud.getRegion());
80 | assertEquals("http://a.com", cloud.getEndpoint());
81 | assertEquals("my-fleet", cloud.getFleet());
82 | assertEquals("my-root", cloud.getFsRoot());
83 | assertTrue(cloud.isPrivateIpUsed());
84 | assertTrue(cloud.isAlwaysReconnect());
85 | assertEquals("myLabel", cloud.getLabelString());
86 | assertEquals(33, cloud.getIdleMinutes());
87 | assertEquals(15, cloud.getMinSize());
88 | assertEquals(90, cloud.getMaxSize());
89 | assertEquals(12, cloud.getNumExecutors());
90 | assertTrue(cloud.isAddNodeOnlyIfRunning());
91 | assertTrue(cloud.isRestrictUsage());
92 | assertEquals(EC2FleetCloud.WeightedScaler.class, cloud.getExecutorScaler().getClass());
93 | assertEquals(181, cloud.getInitOnlineTimeoutSec());
94 | assertEquals(13, cloud.getInitOnlineCheckIntervalSec());
95 | assertEquals(11, cloud.getCloudStatusIntervalSec());
96 | assertTrue(cloud.isDisableTaskResubmit());
97 | assertTrue(cloud.isNoDelayProvision());
98 | assertEquals("xx", cloud.getAwsCredentialsId());
99 |
100 | SSHConnector sshConnector = (SSHConnector) cloud.getComputerConnector();
101 | assertEquals(NonVerifyingKeyVerificationStrategy.class, sshConnector.getSshHostKeyVerificationStrategy().getClass());
102 | }
103 |
104 | @Test
105 | @ConfiguredWithCode("EC2FleetCloud/empty-name-configuration-as-code.yml")
106 | void configurationWithEmptyName_shouldUseDefault(JenkinsConfiguredWithCodeRule jenkinsRule) {
107 | assertEquals(3, jenkinsRule.jenkins.clouds.size());
108 |
109 | for (EC2FleetCloud cloud : jenkinsRule.jenkins.clouds.getAll(EC2FleetCloud.class)){
110 |
111 | assertTrue(cloud.name.startsWith(EC2FleetCloud.BASE_DEFAULT_FLEET_CLOUD_ID));
112 | assertEquals(("FleetCloud".length() + CloudNames.SUFFIX_LENGTH + 1), cloud.name.length());
113 | }
114 | }
115 | }
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/FleetStateStats.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import software.amazon.awssdk.services.ec2.model.BatchState;
4 |
5 | import javax.annotation.Nonnegative;
6 | import javax.annotation.Nonnull;
7 | import javax.annotation.concurrent.ThreadSafe;
8 | import java.util.Map;
9 | import java.util.Objects;
10 | import java.util.Set;
11 |
12 | /**
13 | * @see EC2FleetCloud
14 | */
15 | @SuppressWarnings({"unused"})
16 | @ThreadSafe
17 | public final class FleetStateStats {
18 |
19 | /**
20 | * Abstract state of different implementation of
21 | * {@link com.amazon.jenkins.ec2fleet.fleet.EC2Fleet}
22 | */
23 | public static class State {
24 |
25 | public static State active(final String detailed) {
26 | return new State(true, false, detailed);
27 | }
28 |
29 | public static State modifying(final String detailed) {
30 | return new State(true, true, detailed);
31 | }
32 |
33 | public static State active() {
34 | return active("active");
35 | }
36 |
37 | public static State notActive(final String detailed) {
38 | return new State(false, false, detailed);
39 | }
40 |
41 | private final String detailed;
42 | private final boolean active;
43 | private final boolean modifying;
44 |
45 | public State(final boolean active, final boolean modifying, final String detailed) {
46 | this.detailed = detailed;
47 | this.active = active;
48 | this.modifying = modifying;
49 | }
50 |
51 | /**
52 | * Is underline fleet is updating so we need to suppress update
53 | * until modification will be completed and fleet state will be stabilized.
54 | *
55 | * This is important only for {@link com.amazon.jenkins.ec2fleet.fleet.EC2SpotFleet}
56 | * as it has delay between update request and actual update of target capacity, while
57 | * {@link com.amazon.jenkins.ec2fleet.fleet.AutoScalingGroupFleet} does it in sync with
58 | * update call.
59 | *
60 | * Consumed by {@link EC2FleetCloud#update()}
61 | *
62 | * @return true or false
63 | */
64 | public boolean isModifying() {
65 | return modifying;
66 | }
67 |
68 | /**
69 | * Fleet is good to be used for plugin, it will be shown on UI as option to use
70 | * and plugin will use it for provision {@link EC2FleetCloud#provision(hudson.slaves.Cloud.CloudState, int)} ()} and de-provision
71 | * otherwise activity will be ignored until state will not be updated.
72 | *
73 | * @return true or false
74 | */
75 | public boolean isActive() {
76 | return active;
77 | }
78 |
79 | /**
80 | * Detailed information about EC2 Fleet for example
81 | * EC2 Spot Fleet states are {@link BatchState}
82 | *
83 | * @return string
84 | */
85 | public String getDetailed() {
86 | return detailed;
87 | }
88 |
89 | @Override
90 | public boolean equals(Object o) {
91 | if (this == o) return true;
92 | if (o == null || getClass() != o.getClass()) return false;
93 | State state = (State) o;
94 | return active == state.active &&
95 | Objects.equals(detailed, state.detailed);
96 | }
97 |
98 | @Override
99 | public int hashCode() {
100 | return Objects.hash(detailed, active);
101 | }
102 |
103 | }
104 |
105 | @Nonnull
106 | private final String fleetId;
107 | @Nonnegative
108 | private int numActive;
109 | @Nonnegative
110 | private final int numDesired;
111 | @Nonnull
112 | private final State state;
113 | @Nonnull
114 | private final Set instances;
115 | @Nonnull
116 | private final Map instanceTypeWeights;
117 |
118 | public FleetStateStats(final @Nonnull String fleetId,
119 | final int numDesired, final @Nonnull State state,
120 | final @Nonnull Set instances,
121 | final @Nonnull Map instanceTypeWeights) {
122 | this.fleetId = fleetId;
123 | this.numActive = instances.size();
124 | this.numDesired = numDesired;
125 | this.state = state;
126 | this.instances = instances;
127 | this.instanceTypeWeights = instanceTypeWeights;
128 | }
129 |
130 | public FleetStateStats(final @Nonnull FleetStateStats stats,
131 | final int numDesired) {
132 | this.fleetId = stats.fleetId;
133 | this.numActive = stats.instances.size();
134 | this.numDesired = numDesired;
135 | this.state = stats.state;
136 | this.instances = stats.instances;
137 | this.instanceTypeWeights = stats.instanceTypeWeights;
138 | }
139 |
140 | @Nonnull
141 | public String getFleetId() {
142 | return fleetId;
143 | }
144 |
145 | public int getNumActive() {
146 | return numActive;
147 | }
148 |
149 | // Fleet does not immediately display the active instances and syncs up eventually
150 | public void setNumActive(final int activeCount) {
151 | numActive = activeCount;
152 | }
153 |
154 | public int getNumDesired() {
155 | return numDesired;
156 | }
157 |
158 | @Nonnull
159 | public State getState() {
160 | return state;
161 | }
162 |
163 | @Nonnull
164 | public Set getInstances() {
165 | return instances;
166 | }
167 |
168 | @Nonnull
169 | public Map getInstanceTypeWeights() {
170 | return instanceTypeWeights;
171 | }
172 |
173 | }
174 |
--------------------------------------------------------------------------------
/src/test/java/com/amazon/jenkins/ec2fleet/CloudNamesTest.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import org.junit.jupiter.api.BeforeEach;
4 | import org.junit.jupiter.api.Test;
5 | import org.jvnet.hudson.test.JenkinsRule;
6 | import org.jvnet.hudson.test.junit.jupiter.WithJenkins;
7 |
8 | import static org.junit.jupiter.api.Assertions.*;
9 |
10 | @WithJenkins
11 | class CloudNamesTest {
12 |
13 | private final EC2FleetCloud.ExecutorScaler noScaling = new EC2FleetCloud.NoScaler();
14 |
15 | private JenkinsRule j;
16 |
17 | @BeforeEach
18 | void before(JenkinsRule rule) {
19 | j = rule;
20 | }
21 |
22 | @Test
23 | void isUnique_true() {
24 | j.jenkins.clouds.add(new EC2FleetCloud("SomeDefaultName", null, null, null, null, null,
25 | "test-label", null, null, false, false,
26 | 0, 0, 0, 0, 0, true, false,
27 | "-1", false, 0, 0,
28 | 10, false, false, noScaling));
29 |
30 | assertTrue(CloudNames.isUnique("TestCloud"));
31 | }
32 |
33 | @Test
34 | void isUnique_false() {
35 | j.jenkins.clouds.add(new EC2FleetCloud("SomeDefaultName", null, null, null, null, null,
36 | "test-label", null, null, false, false,
37 | 0, 0, 0, 0, 0, true, false,
38 | "-1", false, 0, 0,
39 | 10, false, false, noScaling));
40 |
41 | assertFalse(CloudNames.isUnique("SomeDefaultName"));
42 | }
43 |
44 | @Test
45 | void isDuplicated_false() {
46 | j.jenkins.clouds.add(new EC2FleetCloud("TestCloud", null, null, null, null, null,
47 | "test-label", null, null, false, false,
48 | 0, 0, 0, 0, 0, true, false,
49 | "-1", false, 0, 0,
50 | 10, false, false, noScaling));
51 |
52 | j.jenkins.clouds.add(new EC2FleetCloud("TestCloud2", null, null, null, null, null,
53 | "test-label", null, null, false, false,
54 | 0, 0, 0, 0, 0, true, false,
55 | "-1", false, 0, 0,
56 | 10, false, false, noScaling));
57 |
58 | assertFalse(CloudNames.isDuplicated("TestCloud"));
59 | }
60 |
61 | @Test
62 | void isDuplicated_true() {
63 | j.jenkins.clouds.add(new EC2FleetCloud("TestCloud", null, null, null, null, null,
64 | "test-label", null, null, false, false,
65 | 0, 0, 0, 0, 0, true, false,
66 | "-1", false, 0, 0,
67 | 10, false, false, noScaling));
68 |
69 | j.jenkins.clouds.add(new EC2FleetCloud("TestCloud", null, null, null, null, null,
70 | "test-label", null, null, false, false,
71 | 0, 0, 0, 0, 0, true, false,
72 | "-1", false, 0, 0,
73 | 10, false, false, noScaling));
74 |
75 | assertTrue(CloudNames.isDuplicated("TestCloud"));
76 | }
77 |
78 | @Test
79 | void generateUnique_noSuffix() {
80 | assertEquals("UniqueCloud", CloudNames.generateUnique("UniqueCloud"));
81 | }
82 |
83 | @Test
84 | void generateUnique_addsSuffixOnlyWhenNeeded() {
85 | j.jenkins.clouds.add(new EC2FleetCloud("UniqueCloud-1", null, null, null, null, null,
86 | "test-label", null, null, false, false,
87 | 0, 0, 0, 0, 0, true, false,
88 | "-1", false, 0, 0,
89 | 10, false, false, noScaling));
90 |
91 | assertEquals("UniqueCloud", CloudNames.generateUnique("UniqueCloud"));
92 | }
93 |
94 | @Test
95 | void generateUnique_addsSuffixCorrectly() {
96 | j.jenkins.clouds.add(new EC2FleetCloud("UniqueCloud", null, null, null, null, null,
97 | "test-label", null, null, false, false,
98 | 0, 0, 0, 0, 0, true, false,
99 | "-1", false, 0, 0,
100 | 10, false, false, noScaling));
101 |
102 | j.jenkins.clouds.add(new EC2FleetCloud("UniqueCloud-1", null, null, null, null, null,
103 | "test-label", null, null, false, false,
104 | 0, 0, 0, 0, 0, true, false,
105 | "-1", false, 0, 0,
106 | 10, false, false, noScaling));
107 |
108 | String actual = CloudNames.generateUnique("UniqueCloud");
109 | assertEquals(actual.length(), ("UniqueCloud".length() + CloudNames.SUFFIX_LENGTH + 1));
110 | assertTrue(actual.startsWith("UniqueCloud-"));
111 | }
112 |
113 | @Test
114 | void generateUnique_emptyStringInConstructor() {
115 | EC2FleetCloud fleetCloud = new EC2FleetCloud("", null, null, null, null, null,
116 | "test-label", null, null, false, false,
117 | 0, 0, 0, 0, 0, true, false,
118 | "-1", false, 0, 0,
119 | 10, false, false, noScaling);
120 |
121 | EC2FleetLabelCloud fleetLabelCloud = new EC2FleetLabelCloud("", null, null,
122 | null, null, new LocalComputerConnector(j), false, false,
123 | 0, 0, 0, 1, false,
124 | false, 0, 0,
125 | 2, false, null);
126 |
127 | assertEquals(("FleetCloud".length() + CloudNames.SUFFIX_LENGTH + 1), fleetCloud.name.length());
128 | assertTrue(fleetCloud.name.startsWith(EC2FleetCloud.BASE_DEFAULT_FLEET_CLOUD_ID));
129 | assertEquals(("FleetLabelCloud".length() + CloudNames.SUFFIX_LENGTH + 1), fleetLabelCloud.name.length());
130 | assertTrue(fleetLabelCloud.name.startsWith(EC2FleetLabelCloud.BASE_DEFAULT_FLEET_CLOUD_ID));
131 | }
132 |
133 | @Test
134 | void generateUnique_nonEmptyStringInConstructor() {
135 | EC2FleetCloud fleetCloud = new EC2FleetCloud("UniqueCloud", null, null, null, null, null,
136 | "test-label", null, null, false, false,
137 | 0, 0, 0, 0, 0, true, false,
138 | "-1", false, 0, 0,
139 | 10, false, false, noScaling);
140 |
141 | EC2FleetLabelCloud fleetLabelCloud = new EC2FleetLabelCloud("UniqueLabelCloud", null, null,
142 | null, null, new LocalComputerConnector(j), false, false,
143 | 0, 0, 0, 1, false,
144 | false, 0, 0,
145 | 2, false, null);
146 |
147 | assertEquals("UniqueCloud", fleetCloud.name);
148 | assertEquals("UniqueLabelCloud", fleetLabelCloud.name);
149 | }
150 | }
151 |
--------------------------------------------------------------------------------
/pom.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 | 4.0.0
4 |
5 |
6 | org.jenkins-ci.plugins
7 | plugin
8 | 5.28
9 |
10 |
11 |
12 | com.amazon.jenkins.fleet
13 | ec2-fleet
14 | ${revision}.${changelist}
15 | hpi
16 |
17 |
18 | 4.2.2
19 | 999999-SNAPSHOT
20 |
21 | 2.492
22 | ${jenkins.baseline}.3
23 | jenkinsci/${project.artifactId}-plugin
24 | false
25 |
26 |
27 | EC2 Fleet Jenkins Plugin
28 | Support EC2 SpotFleet for Jenkins
29 | https://github.com/jenkinsci/${project.artifactId}-plugin
30 |
31 |
32 | MIT License
33 | http://opensource.org/licenses/MIT
34 |
35 |
36 |
37 |
38 | scm:git:https://github.com/${gitHubRepo}.git
39 | scm:git:git@github.com:${gitHubRepo}.git
40 | https://github.com/${gitHubRepo}
41 | ${scmTag}
42 |
43 |
44 |
45 |
46 | repo.jenkins-ci.org
47 | https://repo.jenkins-ci.org/public/
48 |
49 |
50 |
51 |
52 |
53 | repo.jenkins-ci.org
54 | https://repo.jenkins-ci.org/public/
55 |
56 |
57 |
58 |
59 |
60 |
61 | io.jenkins.tools.bom
62 | bom-${jenkins.baseline}.x
63 | 5473.vb_9533d9e5d88
64 | import
65 | pom
66 |
67 |
68 |
69 |
70 |
71 | org.jenkins-ci.plugins
72 | credentials
73 |
74 |
75 | org.jenkins-ci.plugins.workflow
76 | workflow-job
77 | true
78 |
79 |
80 | io.jenkins.plugins.aws-java-sdk2
81 | aws-java-sdk2-core
82 |
83 |
84 | io.jenkins.plugins.aws-java-sdk2
85 | aws-java-sdk2-ec2
86 |
87 |
88 | io.jenkins.plugins.aws-java-sdk2
89 | aws-java-sdk2-autoscaling
90 |
91 |
92 | io.jenkins.plugins.aws-java-sdk2
93 | aws-java-sdk2-cloudformation
94 |
95 |
96 | org.jenkins-ci.plugins
97 | aws-credentials
98 |
99 |
100 | org.jenkins-ci.plugins
101 | ssh-slaves
102 |
103 |
104 |
105 |
106 | org.mockito
107 | mockito-junit-jupiter
108 | test
109 |
110 |
111 | io.jenkins
112 | configuration-as-code
113 | test
114 |
115 |
116 | io.jenkins.configuration-as-code
117 | test-harness
118 | test
119 |
120 |
121 | io.jenkins.plugins.aws-java-sdk2
122 | aws-java-sdk2-iam
123 | test
124 |
125 |
126 |
127 |
128 |
129 |
130 |
131 | org.jenkins-ci.tools
132 | maven-hpi-plugin
133 | true
134 |
135 | 1.45
136 |
137 |
138 |
139 |
140 |
141 |
142 |
143 | jdk17
144 |
145 | [17,)
146 |
147 |
148 |
149 |
150 |
151 |
152 | org.apache.maven.plugins
153 | maven-surefire-plugin
154 |
155 | -Xms768M -Xmx768M -XX:+HeapDumpOnOutOfMemoryError -XX:+TieredCompilation -XX:TieredStopAtLevel=1 @{jenkins.addOpens} @{jenkins.insaneHook} @{jenkins.javaAgent} --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.concurrent.locks=ALL-UNNAMED -Daws.region=us-east-1
156 |
157 |
158 |
159 |
160 |
161 |
162 |
163 |
164 |
--------------------------------------------------------------------------------
/docs/FAQ.md:
--------------------------------------------------------------------------------
1 | ## FAQ
2 |
3 | NOTE: "Jenkins" will refer to the Jenkins code that is not handled by EC2-Fleet-Plugin
4 |
5 | **Q:** How does the EC2-Fleet-Plugin handle scaling?
6 | **A:** For scaling up, the EC2-Fleet-Plugin increases EC2 Fleet's `target capacity` or ASG `desired capacity` to obtain new instances.
7 |
8 | For scaling down, the plugin will manually terminate instances that should be removed and adjust the target/desired capacity.
9 |
10 | **Q:** What's an `update` cycle?
11 | **A:** As long as the plugin is running, the `update` function is called every `Cloud Status Interval` seconds to sync the state of
12 | the plugin with the state of the EC2 Fleet or ASG. Most work done by the plugin occurs within `update`.
13 |
14 | In the logs, "start" denotes the beginning of an `update` cycle.
15 |
16 | **Q:** When does the EC2-Fleet-Plugin scale up?
17 | **A:** When there are pending jobs in the queue, Jenkins will hook into the EC2-Fleet-Plugin to provision new capacity.
18 | If the cloud is able to handle the job (matching labels or no label restrictions), it will calculate how much capacity is needed based on the instance weights and number of executors
19 | per instance. On the next `update` cycle the cloud adjusts `target capacity` to provision new instances. Therefore, if `Cloud Status Interval` is large you will see some delay
20 | between the Jenkins call to provision new instances and the actual API request that changes target capacity.
21 |
22 | **Q:** When does the EC2-Fleet-Plugin scale down?
23 | **A:** Jenkins periodically checks for idle nodes that it can remove. If there are more nodes than `minSize` and either a node has been idle for longer than `Max Idle Minutes Before Scaledown`
24 | or there are more nodes than allowed by `maxSize`, the idle node will be scheduled for termination.
25 |
26 | Pseudo-code:
27 | ```
28 | if (node.isIdle() && numNodes > minSize && (node.isIdleTooLong() || numNodes > maxSize) {
29 | scheduleToTerminate(node);
30 | }
31 | ```
32 |
33 | On the next `update` cycle, an API call is made for each node scheduled for termination. Note that it might be a few more update cycles
34 | before the instances are fully terminated. Check [IdleRetentionStrategy](https://github.com/jenkinsci/ec2-fleet-plugin/blob/master/src/main/java/com/amazon/jenkins/ec2fleet/IdleRetentionStrategy.java)
35 | for details.
36 |
37 | **Q:** What does "first update not done" mean?
38 | **A:** This means that the first `update` cycle hasn't completed and the cloud state is unknown.
39 |
40 | If the plugin configuration was recently saved this shouldn't be a problem, just wait a bit and the update will be triggered after "Cloud Status Interval in sec" seconds.
41 |
42 | If the plugin has been running for a while untouched, there might be an error in the configuration, such as incorrect AWS Credentials. Check the
43 | logs to see if there is any additional information.
44 |
45 | If the plugin version is older than 2.2.2, upgrade to 2.2.2 or later. There was a known bug in older version that was fixed in [#247](https://github.com/jenkinsci/ec2-fleet-plugin/pull/247).
46 |
47 | Otherwise, open an issue and we'll take a look.
48 |
49 | **Q:** Why isn't the plugin scaling down?
50 | **A:** Double check the cloud configuration and the log file. **If `Max Idle Minutes Before Scaledown` is 0, instances will never be removed.**
51 |
52 | If using a plugin version older than 2.2.2, try upgrading. Change [#247](https://github.com/jenkinsci/ec2-fleet-plugin/pull/247)
53 | was released in that version to fix a common problem with instances not being terminated after configuration changes.
54 |
55 | Check the minimum instances on the Fleet or ASG, it might be higher than the min instances in the plugin config.
56 |
57 | If there is nothing abnormal, check for open issues or open a new one if none exist. The plugin should always be able to scale down!
58 |
59 | **Q:** Why isn't the plugin scaling up?
60 | **A:** Double check the cloud configuration and the log file. Also, check the maximum instances on the Fleet or ASG.
61 | If the plugin version is older than 2.2.1, it might be fixed by updating the plugin.
62 | Before that version, modifying the configuration of the plugin during a scaling operation could cause the state of Jenkins and the plugin to become out of sync and require a restart.
63 |
64 | If the plugin version is newer than 2.2.1, check for open issues or open a new one if none exist.
65 |
66 | **Q:** Why does the plugin keep enabling scale-in protection on my ASG?
67 | **A:** The plugin handles termination of instances manually based on idle period settings. Without scale-in protection enabled,
68 | instances could be terminated unexpectedly by external conditions and running jobs could be interrupted.
69 |
70 | **Q:** I only changed one configuration field, why did it reload everything?
71 | **A:** Jenkins doesn't hot swap plugin settings. When 'Save' is clicked, Jenkins will write the plugin configuration to disk and
72 | reinitialize the plugin using the new, current version. If a cloud is modified the plugin will attempt to migrate resources,
73 | but this is not perfect and issues sometimes arise here. If possible, restarting Jenkins after modifying the plugin
74 | configuration often solves most of these problems.
75 |
76 | **Q:** I want to know about _____, but I don't see any information here?
77 | **A:** Check out the [docs](https://github.com/jenkinsci/ec2-fleet-plugin/tree/master/docs) folder. If you're still unable to
78 | find what you're looking for, or you think we should add something, let us know by opening an issue.
79 |
80 | **Q:** Can I contribute?
81 | **A:** Yes, please! Check out the [contributing](https://github.com/jenkinsci/ec2-fleet-plugin/blob/master/CONTRIBUTING.md) page for more information.
82 |
83 | ## Gotchas
84 |
85 | - Modifying the plugin settings will cause all Cloud Fleets to be reconstructed. This can cause strange behavior if done
86 | while jobs are queued or running. If possible, avoid modifying the configuration while jobs are queued or running, or restart
87 | Jenkins after making configuration changes.
88 |
89 | - Max Idle Minutes Before Scaledown is set to 0 so instances are never removed (click the ? on the config page).
90 |
--------------------------------------------------------------------------------
/src/main/resources/com/amazon/jenkins/ec2fleet/EC2FleetCloud/config.jelly:
--------------------------------------------------------------------------------
1 |
2 |
6 |
7 |
8 |
9 |
10 |
11 | A unique name for this EC2 Fleet cloud
12 | Once set, it will be unmodifiable. See this issue for details.
13 |
14 |
15 |
16 |
17 |
18 |
19 |
20 |
21 |
22 |
23 |
24 |
25 |
26 | Select AWS Credentials or leave set to none to use AWS EC2 Instance Role
27 |
28 |
29 |
30 |
31 | Select China region for China credentials.
32 |
33 |
34 |
35 |
36 | Endpoint like https://ec2.us-east-2.amazonaws.com
37 |
38 |
39 |
40 |
41 | Fleet list will be available once region and credentials are specified. Only maintain supported, see help
42 |
43 |
44 |
45 |
46 |
47 |
48 |
49 |
50 |
51 |
52 |
53 |
54 |
55 |
56 | Connect to instances via private IP instead of public IP
57 |
58 |
59 |
60 |
61 | Always reconnect to offline nodes after instance reboot or connection loss
62 |
63 |
64 |
65 |
66 | Only build jobs with label expressions matching this node
67 |
68 |
69 |
70 |
71 |
72 | Labels to add to instances in this fleet
73 |
74 |
75 |
76 |
77 | Default is /tmp/jenkins-<random ID>
78 |
79 |
80 |
81 |
82 | Number of executors per instance
83 |
84 |
85 |
86 |
87 | Method for scaling number of executors
88 |
89 |
90 | How long to keep an idle node. If set to 0, never scale down
91 |
92 |
93 |
94 |
95 |
96 |
97 |
98 |
99 |
100 |
101 |
102 |
103 |
104 |
105 |
106 |
107 |
108 |
110 |
111 |
112 |
113 | Disable auto resubmitting a build if it failed due to an EC2 instance termination like a Spot interruption
114 |
115 |
116 |
117 |
118 | Maximum time to wait for EC2 instance startup
119 |
120 |
121 |
122 |
123 | Interval for updating EC2 cloud status
124 |
125 |
126 |
127 |
128 | Enable faster provision when queue is growing
129 |
130 |
131 |
132 |
133 |
134 |
--------------------------------------------------------------------------------
/src/main/java/com/amazon/jenkins/ec2fleet/EC2FleetAutoResubmitComputerLauncher.java:
--------------------------------------------------------------------------------
1 | package com.amazon.jenkins.ec2fleet;
2 |
3 | import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
4 | import hudson.model.Action;
5 | import hudson.model.Actionable;
6 | import hudson.model.Executor;
7 | import hudson.model.ParametersAction;
8 | import hudson.model.Queue;
9 | import hudson.model.Result;
10 | import hudson.model.TaskListener;
11 | import hudson.model.queue.SubTask;
12 | import hudson.slaves.ComputerLauncher;
13 | import hudson.slaves.DelegatingComputerLauncher;
14 | import hudson.slaves.SlaveComputer;
15 | import org.jenkinsci.plugins.workflow.job.WorkflowJob;
16 | import org.jenkinsci.plugins.workflow.job.WorkflowRun;
17 |
18 | import javax.annotation.concurrent.ThreadSafe;
19 | import java.util.ArrayList;
20 | import java.util.List;
21 | import java.util.logging.Level;
22 | import java.util.logging.Logger;
23 |
24 | /**
25 | * The {@link EC2FleetAutoResubmitComputerLauncher} is responsible for controlling:
26 | * * how {@link EC2FleetNodeComputer}s are launched
27 | * * how {@link EC2FleetNodeComputer}s connect to agents {@link EC2FleetNode}
28 | *
29 | * This is wrapper for {@link ComputerLauncher} to get notification when agent was disconnected
30 | * and automatically resubmit {@link hudson.model.Queue.Task} if reason is unexpected termination
31 | * which usually means EC2 instance was interrupted.
32 | *
33 | * This is optional feature, it's enabled by default, but could be disabled by
34 | * {@link EC2FleetCloud#isDisableTaskResubmit()}
35 | */
36 | @SuppressWarnings("WeakerAccess")
37 | @ThreadSafe
38 | public class EC2FleetAutoResubmitComputerLauncher extends DelegatingComputerLauncher {
39 |
40 | private static final Level LOG_LEVEL = Level.INFO;
41 | private static final Logger LOGGER = Logger.getLogger(EC2FleetAutoResubmitComputerLauncher.class.getName());
42 |
43 | /**
44 | * Delay which will be applied when job {@link Queue#scheduleInternal(Queue.Task, int, List)}
45 | * rescheduled after offline
46 | */
47 | private static final int RESCHEDULE_QUIET_PERIOD_SEC = 10;
48 |
49 | public EC2FleetAutoResubmitComputerLauncher(final ComputerLauncher launcher) {
50 | super(launcher);
51 | }
52 |
53 | /**
54 | * {@link ComputerLauncher#afterDisconnect(SlaveComputer, TaskListener)}
55 | *
56 | * EC2 Fleet plugin overrides this method to detect jobs which were failed because of
57 | * EC2 instance was terminated/stopped. It could be manual stop or because of Spot marked.
58 | * In all cases as soon as job aborted because of broken connection and agent is offline
59 | * it will try to resubmit aborted job back to the queue, so user doesn't need to do that manually
60 | * and another agent could take it.
61 | *
62 | * Implementation details
63 | *
64 | * There is no official recommendation about way how to resubmit job according to
65 | * https://issues.jenkins-ci.org/browse/JENKINS-49707 moreover some of Jenkins code says it impossible.
66 | *