i had updated cdh cluster use spark 1.5.0
. when submit spark application, system show warning spark.app.id
using default name dagscheduler source because spark.app.id not set.
i have searched spark.app.id
not document it. read link , think used restapi call.
i don't see warning in spark 1.4
. explain me , show how set it?
it's not used rest api, rather monitoring purpose e. g when want check yarn logs per example:
yarn logs <spark.app.id>
it's true specific issue still not documented yet. think it's been added standardize application deployment within hadoop ecosystem.
i suggest set 'spark.app.id' in app.
conf.set("spark.app.id", <app-id>) // considering have sparkconf defined of course
nevertheless, remains warning won't effect application itself.
Comments
Post a Comment