Introduction

Introduction

Oracle has introduced an additional schedule on the ODA in patch the 19.16 patch bundles.  The new schedule, think of it as a job, is called SSH cleaner job to clean up stale SSH keys.  Pretty self-explanatory what the schedule does by the name. The new SSH cleaner job schedule is enabled by default and executes every 30 minutes.  The new schedule removes the ida_rsa and id_rsa.pub files for the grid, oracle, and root users. If you didn’t know that this new schedule existed and rely on using SSH keys then like me, things start to break.

Below is my solution with some additional tips on dealing with SSH keys on an ODA.

ODA Schedules

List the available ODA schedules 19.15 vs 19.16.

New for 19.16 a new schedule was introduced,

--19.15 and lower

[root@oda101vm01 ~]# odacli list-schedules

ID                                    Name                      Description                                        CronExpression   Disabled
------------------------------------- ------------------------- -------------------------------------------------- ------------------- -----
1e2eccb9-c275-47c4-97c9-9bb6f0ec419c  Rebalance Monitoring      Rebalance poll and Power modification              0 0/5 * 1/1 * ? *   true
283ca871-3905-441c-a97d-163ae75fe58b  feature_tracking_job      Feature tracking job                               0 0 20 ? * WED *    false
6c4cdf12-6ecf-4d82-893d-a43378a9a467  bom maintenance           bom reports generation                             0 0 1 ? * SUN *     false
c752c67f-c677-40ae-833e-24aafcad9818  Log files Cleanup         Auto log file purge bases on policy                0 0 3 1/1 * ? *     false
fe3b1fc3-0bb3-4ecf-98d6-af4d40a6b0b5  AgentState metastore cleanup internal agentstateentry metastore maintenance  0 0 0 1/1 * ? *     false

--19.16 The SSHCleanerJob is enabled by default (Disabled = false)

[root@oda101vm01 ~]# odacli list-schedules (node 1)

ID                                    Name                      Description                                        CronExpression   Disabled
------------------------------------- ------------------------- -------------------------------------------------- ------------------- -----
1e2eccb9-c275-47c4-97c9-9bb6f0ec419c  Rebalance Monitoring      Rebalance poll and Power modification              0 0/5 * 1/1 * ? *   true
283ca871-3905-441c-a97d-163ae75fe58b  feature_tracking_job      Feature tracking job                               0 0 20 ? * WED *    false
6c4cdf12-6ecf-4d82-893d-a43378a9a467  bom maintenance           bom reports generation                             0 0 1 ? * SUN *     false
b0b28833-80ee-4ff8-8a8a-0bbd4949ea63  SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *  false
c752c67f-c677-40ae-833e-24aafcad9818  Log files Cleanup         Auto log file purge bases on policy                0 0 3 1/1 * ? *     false
fe3b1fc3-0bb3-4ecf-98d6-af4d40a6b0b5  AgentState metastore cleanup internal agentstateentry metastore maintenance  0 0 0 1/1 * ? *     false

[root@oda102vm01 ~]#  odacli list-schedules (node 2)

ID                                   Name                      Description                                        CronExpression     Disabled
------------------------------------ ------------------------- -------------------------------------------------- ------------------ --------
5a1aa639-4c6a-4d96-bb02-d3ed4e2b2245 feature_tracking_job      Feature tracking job                               0 0 20 ? * WED *   false
704e2f3f-ef41-4b81-8246-d062203015ea bom maintenance           bom reports generation                             0 0 1 ? * SUN *    false
b0aa90ca-f847-47c1-beef-88bc42306c11 AgentState metastore cleanup internal agentstateentry metastore maintenance  0 0 0 1/1 * ? *    false
be68182f-81e1-41c8-9900-89e7d7900d52 SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? * false
ec0848d0-a17a-40ca-9c8f-16810b701fe1 Log files Cleanup         Auto log file purge bases on policy                0 0 3 1/1 * ? *    false
f2e6c810-01ac-40f7-9c0e-21905d67df45 Rebalance Monitoring      Rebalance poll and Power modification              0 0/5 * 1/1 * ? *  true

Disable the SSHCleanerJob schedule.

The schedules are unique to each node of the bare metal of DBSystem cluster. You will want to run this on both nodes.

--If desired disable the SSHKey cleanup job

[root@oda101vm02 ~]# odacli update-schedule -i b0b28833-80ee-4ff8-8a8a-0bbd4949ea63 -d
Update job schedule success

[root@oda101vm01 ~]# odacli list-schedules (node 1)

ID                                    Name                      Description                                        CronExpression   Disabled
------------------------------------- ------------------------- -------------------------------------------------- ------------------- -----
1e2eccb9-c275-47c4-97c9-9bb6f0ec419c  Rebalance Monitoring      Rebalance poll and Power modification              0 0/5 * 1/1 * ? *   true
283ca871-3905-441c-a97d-163ae75fe58b  feature_tracking_job      Feature tracking job                               0 0 20 ? * WED *    false
6c4cdf12-6ecf-4d82-893d-a43378a9a467  bom maintenance           bom reports generation                             0 0 1 ? * SUN *     false
b0b28833-80ee-4ff8-8a8a-0bbd4949ea63  SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? *  true
c752c67f-c677-40ae-833e-24aafcad9818  Log files Cleanup         Auto log file purge bases on policy                0 0 3 1/1 * ? *     false
fe3b1fc3-0bb3-4ecf-98d6-af4d40a6b0b5  AgentState metastore cleanup internal agentstateentry metastore maintenance  0 0 0 1/1 * ? *     false


[root@oda102vm01 ~]#  odacli list-schedules (node 2)

ID                                   Name                      Description                                        CronExpression     Disabled
------------------------------------ ------------------------- -------------------------------------------------- ------------------ --------
5a1aa639-4c6a-4d96-bb02-d3ed4e2b2245 feature_tracking_job      Feature tracking job                               0 0 20 ? * WED *   false
704e2f3f-ef41-4b81-8246-d062203015ea bom maintenance           bom reports generation                             0 0 1 ? * SUN *    false
b0aa90ca-f847-47c1-beef-88bc42306c11 AgentState metastore cleanup internal agentstateentry metastore maintenance  0 0 0 1/1 * ? *    false
be68182f-81e1-41c8-9900-89e7d7900d52 SSHCleanerJob             SSH cleaner job to clean up stale SSH keys         0 0/30 * 1/1 * ? * true
ec0848d0-a17a-40ca-9c8f-16810b701fe1 Log files Cleanup         Auto log file purge bases on policy                0 0 3 1/1 * ? *    false
f2e6c810-01ac-40f7-9c0e-21905d67df45 Rebalance Monitoring      Rebalance poll and Power modification              0 0/5 * 1/1 * ? *  true

Review the SSHCleanerJob schedule details

To review additional details of the schedule.

--review the odacli help for describing a schedule.

[root@oda101vm01 ~]# odacli  describe-schedule -h
describe-schedule
    Describes a specific schedule

Syntax
    describe-schedule -i  [-j] [-h]

Parameters
    Options                    Description
    -------------------------  -------------------------------------------------
    --scheduleid,-i (*)        Schedule ID

    --json,-j                  Specifies to show output in JSON format

    --help,-h                  Specifies to display help message

[root@oda101vm01 ~]# odacli describe-schedule -i b0b28833-80ee-4ff8-8a8a-0bbd4949ea63

Job Schedule details
----------------------------------------------------------------
                     ID:  b0b28833-80ee-4ff8-8a8a-0bbd4949ea63
                JobName:  SSHCleanerJob
               JobGroup:  SSHCleanerJobGroup
         CronExpression:  0 0/30 * 1/1 * ? *
               JobClass:  com.oracle.dcs.agent.schedule.jobs.SSHCleanerJob
            UpdatedTime:  November 25, 2022 4:50:55 PM EST
            Description:  SSH cleaner job to clean up stale SSH keys
                Disable:  true

Workaround to DISABLING the SSHCleanerJOB Schedule

Work Around to stop the SSHCleanerJob schedule from removing the SSH key files.

Initially, I didn’t realize that the SSHCleanerJob was introduced and needed a quick workaround to stop the SSH keys from being deleted.  I made the id_rsa and id_rsa.pub files to be immutable. I reverted this once I disabled the SSHCleanerJob schedules.  I do not recommend this as a permanent solution.

--ssh key lockdown (make immutable) as the root user on each node.
chattr +i -V /root/.ssh/id_rsa*
chattr +i -V /home/grid/.ssh/id_rsa*
chattr +i -V /home/oracle/.ssh/id_rsa*

--ssh key unlock (required to make any changes to these files.)
chattr -i /root/.ssh/id_rsa*
chattr -i /home/grid/.ssh/id_rsa*
chattr -i /home/oracle/.ssh/id_rsa*

RECOMMENDATION – BACKUP SSH KEY files

Back up your SSH key files.

As part of the ODA patching steps, the SSH key files are removed and not recreated.  This can be another frustrating feature that happens when patching. To counter this I recommend backing up your SSH keys and authorized_key file as a pre-ODA patching step.

--perform the following for root, oracle, and grid users on both nodes

--root
cp -p ~/.ssh/id_rsa /backup_location/`whoami`.`uname -n`.id_rsa
cp -p ~/.ssh/id_rsa.pub /backup_location/`whoami`.`uname -n`.id_rsa.pub
cp -p ~/.ssh/authorized_keys /backup_location/`whoami`.`uname -n`. authorized_keys

--grid
cp -p ~/.ssh/id_rsa /backup_location/`whoami`.`uname -n`.id_rsa
cp -p ~/.ssh/id_rsa.pub /backup_location/`whoami`.`uname -n`.id_rsa.pub
cp -p ~/.ssh/authorized_keys /backup_location/`whoami`.`uname -n`. authorized_keys

--oracle
cp -p ~/.ssh/id_rsa /backup_location/`whoami`.`uname -n`.id_rsa
cp -p ~/.ssh/id_rsa.pub /backup_location/`whoami`.`uname -n`.id_rsa.pub
cp -p ~/.ssh/authorized_keys /backup_location/`whoami`.`uname -n`. authorized_keys

Restore your SSH key files.

Once ODA patching has been completed, restore the original SSH keys and authorized_keys file.

--perform the following for root, oracle, and grid users on both nodes keeping the original file timestamp.

--root
cp -p /backup_location/`whoami`.`uname -n`.id_rsa ~/.ssh/id_rsa
cp -p /backup_location/`whoami`.`uname -n`.id_rsa.pub ~/.ssh/id_rsa.pub
cp -p /backup_location/`whoami`.`uname -n`.authorized_keys ~/.ssh/authorized_keys

--grid
cp -p /backup_location/`whoami`.`uname -n`.id_rsa ~/.ssh/id_rsa
cp -p /backup_location/`whoami`.`uname -n`.id_rsa.pub ~/.ssh/id_rsa.pub
cp -p /backup_location/`whoami`.`uname -n`.authorized_keys ~/.ssh/authorized_keys

--oracle
cp -p /backup_location/`whoami`.`uname -n`.id_rsa ~/.ssh/id_rsa
cp -p /backup_location/`whoami`.`uname -n`.id_rsa.pub ~/.ssh/id_rsa.pub
cp -p /backup_location/`whoami`.`uname -n`.authorized_keys ~/.ssh/authorized_keys

CONCLUSION

Oracle sets the rules.

Typically when Oracle introduces new functionality there is a good reason why. Sometimes this isn’t the case such as in the SSH cleaner job to clean up job.

Verify with Oracle Support.

I did open an SR with Oracle support to ensure that I wasn’t doing something that I shouldn’t be doing. Oracle confirmed that it is allowed to disable the SSH cleaner job as you see fit. Referencing Documents Patching Oracle Database Appliance Oracle Database Appliance Command-Line Interface

 

Referencing Documents

 

Patching Oracle Database Appliance

Oracle Database Appliance Command-Line Interface