Greetings! I'm new to puppet enterprise, and trying to get up and running with a new control repository.
Mostly this works fine, but sometimes after a restart of service pe-puppetserver, I am unable to deploy with code manager. When I run, via the command line, `puppet-code deploy --all --wait -l trace`, I get the following output:
```
2016-11-01 20:01:18.775517 DEBUG puppetlabs.puppet-code - Loaded global configuration from /etc/puppetlabs/client-tools/puppet-code.conf
2016-11-01 20:01:18.776012 DEBUG leatherman.file_util:61 - Error reading file: No such file or directory
2016-11-01 20:01:18.776127 DEBUG puppetlabs.puppet-code - Could not read user configuration from /root/.puppetlabs/client-tools/puppet-code.conf: file not present or readable.
Deploying all environments.
2016-11-01 20:01:18.776416 DEBUG puppetlabs.puppet-access - Reading token at path /root/.puppetlabs/token
2016-11-01 20:01:18.779199 DEBUG puppetlabs.puppet-code - puppet-code version is 1.2.0 (commit da7522fa66bb4167e60c8953a5060d8a897ac68d)
2016-11-01 20:01:18.779336 DEBUG leatherman.curl:223 - requesting https://puppetmaster.uss.net:8170/code-manager/v1/deploys.
2016-11-01 20:01:18.784164 DEBUG leatherman.curl:463 - Trying 10.3.29.70...
2016-11-01 20:01:18.784226 DEBUG leatherman.curl:463 - TCP_NODELAY set
2016-11-01 20:01:18.784433 DEBUG leatherman.curl:463 - Connected to puppetmaster.uss.net (10.3.29.70) port 8170 (#0)
2016-11-01 20:01:18.784771 DEBUG leatherman.curl:463 - ALPN, offering http/1.1
2016-11-01 20:01:18.784868 DEBUG leatherman.curl:463 - Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
2016-11-01 20:01:18.785029 DEBUG leatherman.curl:463 - successfully set certificate verify locations:
2016-11-01 20:01:18.785077 DEBUG leatherman.curl:463 - CAfile: /etc/puppetlabs/puppet/ssl/certs/ca.pem
CApath: none
2016-11-01 20:01:18.785223 DEBUG leatherman.curl:463 - TLSv1.2 (OUT), TLS header, Certificate Status (22):
2016-11-01 20:01:18.785267 DEBUG leatherman.curl:463 - TLSv1.2 (OUT), TLS handshake, Client hello (1):
2016-11-01 20:01:18.800199 DEBUG leatherman.curl:463 - TLSv1.2 (IN), TLS handshake, Server hello (2):
2016-11-01 20:01:18.800310 DEBUG leatherman.curl:463 - TLSv1.2 (IN), TLS handshake, Certificate (11):
2016-11-01 20:01:18.800986 DEBUG leatherman.curl:463 - TLSv1.2 (IN), TLS handshake, Request CERT (13):
2016-11-01 20:01:18.801074 DEBUG leatherman.curl:463 - TLSv1.2 (IN), TLS handshake, Server finished (14):
2016-11-01 20:01:18.801119 DEBUG leatherman.curl:463 - TLSv1.2 (OUT), TLS handshake, Certificate (11):
2016-11-01 20:01:18.801612 DEBUG leatherman.curl:463 - TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
2016-11-01 20:01:18.801654 DEBUG leatherman.curl:463 - TLSv1.2 (OUT), TLS change cipher, Client hello (1):
2016-11-01 20:01:18.801792 DEBUG leatherman.curl:463 - TLSv1.2 (OUT), TLS handshake, Finished (20):
2016-11-01 20:01:18.863409 DEBUG leatherman.curl:463 - TLSv1.2 (IN), TLS change cipher, Client hello (1):
2016-11-01 20:01:18.863568 DEBUG leatherman.curl:463 - TLSv1.2 (IN), TLS handshake, Finished (20):
2016-11-01 20:01:18.863627 DEBUG leatherman.curl:463 - SSL connection using TLSv1.2 / AES256-SHA256
2016-11-01 20:01:18.863661 DEBUG leatherman.curl:463 - ALPN, server did not agree to a protocol
2016-11-01 20:01:18.863702 DEBUG leatherman.curl:463 - Server certificate:
2016-11-01 20:01:18.863750 DEBUG leatherman.curl:463 - subject: CN=puppetmaster.uss.net
2016-11-01 20:01:18.863788 DEBUG leatherman.curl:463 - start date: Oct 23 15:08:54 2016 GMT
2016-11-01 20:01:18.863817 DEBUG leatherman.curl:463 - expire date: Oct 23 15:08:54 2021 GMT
2016-11-01 20:01:18.863868 DEBUG leatherman.curl:463 - subjectAltName: host "puppetmaster.uss.net" matched cert's "puppetmaster.uss.net"
2016-11-01 20:01:18.863909 DEBUG leatherman.curl:463 - issuer: CN=Puppet Enterprise CA generated on puppetmaster.uss.net at +2016-10-24 23:08:46 +0800
2016-11-01 20:01:18.863939 DEBUG leatherman.curl:463 - SSL certificate verify ok.
2016-11-01 20:01:18.864111 TRACE leatherman.curl:479 - [request headers: 1167 bytes]
POST /code-manager/v1/deploys HTTP/1.1
Host: puppetmaster.uss.net:8170
Accept: */*
Cookie:
Accept-Language: *
Content-Type: application/json
User-Agent: CodeManagerCLI/1.2.0 (commit da7522fa66bb4167e60c8953a5060d8a897ac68d)
X-Authentication: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJjb2RlbWFuYWdlciIsImlhdCI6MTQ3Nzk5MTkxMSwiZXhwIjoxNDc4MDc4MzExLCJzdWIiOnsiaWQiOiJkNGY3OTk1OC1jNmIwLTRjZDEtOWMyZC03YjFlNzI0YWYyNmUiLCJsb2dpbiI6ImNvZGVtYW5hZ2VyIn19.NE2XkbaTjJKv
8MEG7Vw9pPk7R8HxtL7RcYFeKAb-uzmDES-TxXuf0_qVVn2My-jhkHS8T2DwYuX1U1kM8s0UUdNE-JI6S4nJMssFC9YZAkmI6SjbdORGt2MDGUv_eANGZz_s-e08E8sQvlXYGJBhHJq5-BF0hDMP__Atb3v7wLS_02OZIpdUgGnUGebZ8Ixv-p5jDvPS1H7fQ7faV5vN3iv0ctrKkEtfrHZWoklWC5hUXCLnyMzQhJR43pBG
_Sv7QqP78eFGqGRSZpPy8XSfXfI4Ra07DRa-ZdQNWfFEZdkkhLLnw91t1alZzSO1rcAIaZYjtjR-luiumGjonY3Th3d4juWB4j8soO5jDZv-Op8PQJFUjowS7O5UEMy5mLBlu4v4FJF92-IWaDv3OsaaSIf6jr3oDilsyieXZiYJiDHhOgmpYSV10of3FP7NosQBXtb2kI_fw4iTq1277tNXTidD7WEdpnmjgNdJiebMoBl3
ryvktHVINING_bRrBmQSMAejqlBn6eFm-0cmfDAPSRMnUvX82EKuhy_QZqEv4hP_QLOTASrwcVUgasmzRExj861-bT-zjHATCw90z43dNNO4otZeeA9bXBy9AOYueJHkfcmwSVBsLgl-eyECpKsOSpFmt8amQwtOJtk9hVVE_3doNQZfRoLsXieckH6Tvuc
Content-Length: 31
2016-11-01 20:01:18.864230 TRACE leatherman.curl:479 - [request body: 31 bytes]
{"deploy-all":true,"wait":true}
2016-11-01 20:01:18.864267 DEBUG leatherman.curl:463 - We are completely uploaded and fine
```
The console will hang there for a long time before finally finishing the request with the lines:
```
2016-11-01 20:03:56.454577 TRACE leatherman.curl:479 - [response headers: 27 bytes]
HTTP/1.1 500 Server Error
2016-11-01 20:03:56.454740 TRACE leatherman.curl:479 - [response headers: 37 bytes]
Date: Tue, 01 Nov 2016 12:01:18 GMT
2016-11-01 20:03:56.454820 TRACE leatherman.curl:479 - [response headers: 32 bytes]
Content-Type: application/json
2016-11-01 20:03:56.454888 TRACE leatherman.curl:479 - [response headers: 21 bytes]
Content-Length: 118
2016-11-01 20:03:56.454956 TRACE leatherman.curl:479 - [response headers: 31 bytes]
Server: Jetty(9.2.z-SNAPSHOT)
2016-11-01 20:03:56.455015 TRACE leatherman.curl:479 - [response body: 118 bytes]
{"kind": "internal-server-error", "msg": "An error occured while attempting to process your request", "details": null}
2016-11-01 20:03:56.455072 DEBUG leatherman.curl:463 - Curl_http_done: called premature == 0
2016-11-01 20:03:56.455132 DEBUG leatherman.curl:463 - Connection #0 to host puppetmaster.uss.net left intact
2016-11-01 20:03:56.455199 DEBUG leatherman.curl:165 - request completed (status 500).
{"kind":"internal-server-error","msg":"An error occured while attempting to process your request","details":null}
```
When I inspect the log file at `/var/log/puppetlabs/puppetserver/puppetserver.log`, I can see the following java.lang exception thrown:
```
2016-11-01 20:03:56,450 ERROR [qtp1396103453-76] [p.c.app] Error serving request: /code-manager/v1/deploys
java.lang.IllegalArgumentException: Cannot format given Object as a Number
at java.text.DecimalFormat.format(DecimalFormat.java:507)
at java.text.Format.format(Format.java:157)
at java.text.MessageFormat.subformat(MessageFormat.java:1322)
at java.text.MessageFormat.format(MessageFormat.java:865)
at java.text.Format.format(Format.java:157)
at puppetlabs.i18n.core$fmt.invoke(core.clj:236)
at puppetlabs.i18n.core$translate.doInvoke(core.clj:243)
at clojure.lang.RestFn.invoke(RestFn.java:467)
at puppetlabs.code_manager.app$wrap_format_errors$format_errors__29876.invoke(app.clj:160)
at puppetlabs.code_manager.app$wrap_add_version_header$add_version_header__29901.invoke(app.clj:193)
at ring.middleware.json$wrap_json_response$fn__26855.invoke(json.clj:65)
at puppetlabs.code_manager.app$wrap_catch_500$catch_500__29895.invoke(app.clj:182)
at puppetlabs.i18n.core$locale_negotiator$fn__110.invoke(core.clj:343)
at compojure.response$eval22904$fn__22905.invoke(response.clj:33)
at compojure.response$eval22871$fn__22872$G__22862__22879.invoke(response.clj:6)
at puppetlabs.code_manager.app$build_routes$fn__29906.invoke(app.clj:216)
at bidi.ring$eval22685$fn__22686.invoke(ring.clj:21)
at bidi.ring$eval22664$fn__22665$G__22655__22674.invoke(ring.clj:12)
at puppetlabs.comidi$make_handler$fn__24699.invoke(comidi.clj:249)
at puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__40593.invoke(jetty9_core.clj:424)
at puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
```
Any idea what's going on here? This has sort of magically resolved itself the last few times it has happened. This time round the error does not seem to be going anywhere. Any advice on how to proceed would be greatly appreciated!
Puppet Enterprise version is v2016.4.0
↧